Nov 03 21:14:19.151919 ip-10-0-128-179 systemd[1]: Starting Kubernetes Kubelet... Nov 03 21:14:19.536836 ip-10-0-128-179 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:14:19.536836 ip-10-0-128-179 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 21:14:19.536836 ip-10-0-128-179 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:14:19.536836 ip-10-0-128-179 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:14:19.539829 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539744 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.539829 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539829 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.539829 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539833 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539836 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539839 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539842 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539845 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539849 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539851 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539854 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539856 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539859 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539862 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539865 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539867 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539870 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539872 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539875 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539878 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539880 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539883 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539886 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.539918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539888 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539891 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539893 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539896 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539899 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539901 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539904 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539910 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539913 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539915 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539918 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539921 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539924 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539929 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539933 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539937 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539940 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539943 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539945 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.540376 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539948 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539950 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539953 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539956 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539959 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539962 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539964 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539967 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539969 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539972 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539974 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539976 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539981 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539985 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539988 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539991 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539994 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539997 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.539999 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.540833 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540002 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540004 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540007 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540010 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540013 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540016 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540019 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540022 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540024 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540027 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540029 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540032 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540034 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540037 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540039 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540042 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540044 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540047 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540050 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.541276 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540052 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540056 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540059 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540061 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540063 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540066 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540068 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540071 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540073 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540076 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540079 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540084 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540086 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540089 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540091 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540094 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540096 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540099 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540102 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540105 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.541736 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540108 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540110 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540113 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540115 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540118 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540121 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540124 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540883 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540890 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540893 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540896 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540898 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540902 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540904 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540907 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540910 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540913 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540915 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540918 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540921 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.542204 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540923 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540926 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540928 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540931 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540935 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540939 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540941 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540945 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540948 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540950 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540953 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540956 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540958 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540961 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540964 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540966 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540969 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540972 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540974 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540977 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.542723 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540980 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540982 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540985 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540988 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540990 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540993 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540996 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.540999 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541001 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541004 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541008 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541012 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541015 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541017 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541020 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541022 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541025 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541027 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.543194 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541030 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541032 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541035 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541038 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541041 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541043 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541046 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541049 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541051 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541054 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541057 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541059 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541062 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541064 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541067 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541069 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541072 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541074 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541077 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541079 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.544125 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541082 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541084 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541095 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541098 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541100 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541103 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541105 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541108 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541111 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541113 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541116 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541118 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541121 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541123 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541126 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541128 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541131 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541134 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541137 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541140 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.544651 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541143 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541146 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541148 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541151 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541153 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541156 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541158 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541161 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541163 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541166 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541168 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541171 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541173 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541176 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541178 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541687 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541693 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541696 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541700 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.545118 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541703 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541705 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541708 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541711 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541717 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541720 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541723 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541726 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541729 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541732 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541735 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541737 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541741 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541743 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541746 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541749 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541752 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541754 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541757 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.545556 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541759 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541762 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541764 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541767 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541769 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541772 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541774 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541777 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541779 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541782 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541785 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541788 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541790 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541793 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541795 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541798 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541800 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541803 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541805 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541808 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.546011 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541811 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541813 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541821 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541825 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541828 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541831 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541834 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541836 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541839 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541841 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541844 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541846 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541849 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541851 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541854 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541856 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541859 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541862 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541864 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.546458 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541867 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541869 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541872 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541875 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541878 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541880 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541885 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541889 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541892 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541894 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541897 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541899 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541902 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541905 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541907 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541911 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541914 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541916 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541919 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.546918 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541922 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541924 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541927 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541929 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541932 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541935 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541937 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541940 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541942 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541945 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541947 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541950 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541952 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541954 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541957 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541960 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541962 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541965 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541968 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541970 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.547337 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541973 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541976 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541979 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541981 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.541984 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542908 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542918 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542924 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542930 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542934 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542938 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542942 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542946 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542949 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542952 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542956 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542959 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542962 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542965 2575 flags.go:64] FLAG: --cgroup-root="" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542967 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542970 2575 flags.go:64] FLAG: --client-ca-file="" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542973 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542976 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542981 2575 flags.go:64] FLAG: --cluster-domain="" Nov 03 21:14:19.547926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542984 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542987 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542990 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542993 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.542997 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543000 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543003 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543007 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543009 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543012 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543015 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543018 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543021 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543025 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543030 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543033 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543037 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543040 2575 flags.go:64] FLAG: --enable-server="true" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543042 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543046 2575 flags.go:64] FLAG: --event-burst="100" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543049 2575 flags.go:64] FLAG: --event-qps="50" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543052 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543055 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543058 2575 flags.go:64] FLAG: --eviction-hard="" Nov 03 21:14:19.548478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543062 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543066 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543069 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543072 2575 flags.go:64] FLAG: --eviction-soft="" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543075 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543077 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543080 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543083 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543086 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543089 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543091 2575 flags.go:64] FLAG: --feature-gates="" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543095 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543098 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543101 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543104 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543108 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543111 2575 flags.go:64] FLAG: --help="false" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543113 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543117 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543119 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543122 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543125 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543132 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 21:14:19.549098 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543135 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543138 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543141 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543144 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543147 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543150 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543152 2575 flags.go:64] FLAG: --kube-reserved="" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543155 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543158 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543162 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543164 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543167 2575 flags.go:64] FLAG: --lock-file="" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543170 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543173 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543176 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543181 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543184 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543187 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543190 2575 flags.go:64] FLAG: --logging-format="text" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543192 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543196 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543199 2575 flags.go:64] FLAG: --manifest-url="" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543202 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543206 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543209 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 21:14:19.549851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543213 2575 flags.go:64] FLAG: --max-pods="110" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543216 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543219 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543222 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543225 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543228 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543232 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543235 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543243 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543246 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543249 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543252 2575 flags.go:64] FLAG: --pod-cidr="" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543255 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543257 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543260 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543263 2575 flags.go:64] FLAG: --port="10250" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543266 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543269 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-08b57ca8d6ac47f41" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543272 2575 flags.go:64] FLAG: --qos-reserved="" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543274 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543278 2575 flags.go:64] FLAG: --register-node="true" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543280 2575 flags.go:64] FLAG: --register-with-taints="" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543284 2575 flags.go:64] FLAG: --registry-burst="10" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543287 2575 flags.go:64] FLAG: --registry-qps="5" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543289 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543292 2575 flags.go:64] FLAG: --reserved-memory="" Nov 03 21:14:19.550895 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543296 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543299 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543302 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543305 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543307 2575 flags.go:64] FLAG: --runonce="false" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543310 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543313 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543317 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543320 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543323 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543326 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543330 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543335 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543337 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543340 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543343 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543346 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543349 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543352 2575 flags.go:64] FLAG: --system-cgroups="" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543355 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543360 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543363 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543366 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543370 2575 flags.go:64] FLAG: --tls-min-version="" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543373 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 21:14:19.551995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543376 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543379 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543381 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543384 2575 flags.go:64] FLAG: --v="2" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543389 2575 flags.go:64] FLAG: --version="false" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543393 2575 flags.go:64] FLAG: --vmodule="" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543397 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543400 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543528 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543532 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543534 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543537 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543540 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543542 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543546 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543549 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543551 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543554 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543558 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543562 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543565 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543567 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.553054 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543570 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543572 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543575 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543577 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543580 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543582 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543585 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543588 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543590 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543593 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543596 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543598 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543601 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543603 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543606 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543621 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543624 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543626 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543629 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543631 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.553981 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543634 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543637 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543641 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543644 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543647 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543650 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543653 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543656 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543660 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543663 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543666 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543668 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543671 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543674 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543676 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543679 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543681 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543684 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543687 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.554909 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543689 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543692 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543694 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543697 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543700 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543702 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543705 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543707 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543710 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543712 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543715 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543717 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543720 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543722 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543724 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543727 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543729 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543732 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543734 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.555380 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543738 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543742 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543747 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543751 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543754 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543756 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543759 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543761 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543764 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543766 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543769 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543772 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543775 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543777 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543780 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543782 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543785 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543787 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543790 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.556046 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543793 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543795 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543798 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543800 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543803 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543805 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543807 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543810 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543813 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543815 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543818 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543820 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543823 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543825 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543828 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.543855 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:14:19.556788 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543895 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543898 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543902 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543905 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543908 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543911 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543914 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543917 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543919 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543922 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543925 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543927 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543930 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543932 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543935 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543937 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543940 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543943 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543945 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543947 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543950 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.557332 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543952 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543955 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543957 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543960 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543962 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543965 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543967 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543970 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543972 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543975 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543978 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543985 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543988 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543991 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543994 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543996 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.543999 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544002 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544004 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544007 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.558029 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544009 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544012 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544014 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544017 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544019 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544022 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544024 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544027 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544029 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544032 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544034 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544037 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544039 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544042 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544044 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544046 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544049 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544051 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544054 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.558692 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544056 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544059 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544061 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544064 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544069 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544072 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544074 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544078 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544082 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544085 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544088 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544091 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544093 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544095 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544098 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544100 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544103 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544106 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.559350 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544110 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544113 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544115 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544118 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544120 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544123 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544125 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544128 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544131 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544134 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544137 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544140 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544142 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544145 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544147 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544150 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544152 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544155 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544159 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.559867 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544162 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544165 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544167 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544170 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544173 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544176 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544178 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544181 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.544183 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.552655 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.552676 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.552694 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.552702 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552801 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552809 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552814 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552820 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552824 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552828 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.560327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552832 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552837 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552841 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552845 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552849 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552853 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552860 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552866 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552871 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552876 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552880 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552884 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552889 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552894 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552898 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552902 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552906 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552911 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552915 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.560888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552920 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552924 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552928 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552932 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552946 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552951 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552955 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552959 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552964 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552968 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552972 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552976 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552981 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552985 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552989 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552994 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.552998 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553002 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553006 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553010 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.561327 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553015 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553020 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553024 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553029 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553033 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553037 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553041 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553045 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553049 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553053 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553057 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553061 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553065 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553068 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553072 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553076 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553080 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553093 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553097 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.561809 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553101 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553105 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553110 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553114 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553118 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553122 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553126 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553131 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553135 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553139 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553143 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553147 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553151 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553156 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553160 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553164 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553169 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553173 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553177 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553182 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.562271 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553186 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553190 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553194 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553198 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553202 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553206 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553210 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553215 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553219 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553223 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553227 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553241 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553248 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553253 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553259 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553265 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553269 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553273 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553277 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.562747 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553281 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553285 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553289 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.553330 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553639 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553648 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553653 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553658 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553663 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553667 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553671 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553676 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553680 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553684 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553688 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553693 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.563173 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553697 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553703 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553717 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553722 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553727 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553731 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553735 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553739 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553743 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553754 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553759 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553763 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553767 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553771 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553775 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553779 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553784 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553788 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553792 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553796 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.563572 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553800 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553804 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553808 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553813 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553817 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553821 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553825 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553829 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553834 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553838 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553842 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553846 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553850 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553854 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553858 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553862 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553866 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553870 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553874 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553878 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.564151 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553882 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553886 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553898 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553902 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553906 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553910 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553914 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553918 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553922 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553927 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553931 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553935 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553939 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553943 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553948 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553952 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553956 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553960 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553965 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.564636 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553969 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553973 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553977 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553981 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553985 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553988 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553994 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.553999 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554004 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554009 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554013 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554018 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554022 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554026 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554030 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554034 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554048 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554052 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554056 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.565207 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554061 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554064 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554068 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554072 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554077 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554081 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554088 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554092 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554097 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554101 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554105 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554109 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554113 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554117 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554121 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554125 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.565679 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.554164 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554225 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554233 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554238 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554242 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554246 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554250 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554255 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554259 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554263 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554267 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554271 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554276 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554280 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554292 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554296 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554300 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554304 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554308 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554312 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:14:19.566072 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554316 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554320 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554325 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554331 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554335 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554340 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554344 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554348 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554353 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554357 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554361 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554365 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554369 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554373 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554377 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554381 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554387 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554393 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554398 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:14:19.566606 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554402 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554406 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554410 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554414 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554419 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554423 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554427 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554431 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554443 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554447 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554451 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554455 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554459 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554464 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554468 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554472 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554477 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554482 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554486 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554490 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:14:19.567251 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554494 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554498 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554503 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554507 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554511 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554515 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554519 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554524 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554528 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554532 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554536 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554541 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554545 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554549 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554553 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554557 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554561 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554565 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554569 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:14:19.567882 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554574 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554578 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554589 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554593 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554597 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554601 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554625 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554630 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554633 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554639 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554644 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554648 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554652 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554655 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554659 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554663 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554667 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554671 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554675 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:14:19.568339 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554678 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554682 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554686 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554692 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554698 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554703 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554707 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554711 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554715 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:19.554719 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.554810 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.560522 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.561358 2575 server.go:1014] "Starting client certificate rotation" Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.561448 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.561481 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.563386 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:14:19.568942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.566114 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:14:19.575345 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.575329 2575 log.go:25] "Validated CRI v1 runtime API" Nov 03 21:14:19.578101 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.578083 2575 log.go:25] "Validated CRI v1 image API" Nov 03 21:14:19.578907 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.578886 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 21:14:19.584144 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.584121 2575 fs.go:135] Filesystem UUIDs: map[37c83c4d-ce9f-4d61-a220-ca63fe66b4c1:/dev/nvme0n1p3 799d1d18-ac6b-4e44-9d3d-5913512821c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 21:14:19.584205 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.584143 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 21:14:19.590207 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.590100 2575 manager.go:217] Machine: {Timestamp:2026-11-03 21:14:19.587767204 +0000 UTC m=+0.321030968 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3195369 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c8162b6e8fefb291cb2ad7da3da7f SystemUUID:ec2c8162-b6e8-fefb-291c-b2ad7da3da7f BootID:2f33450a-8758-4c3e-8fca-e06c97bcc736 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0d:71:0f:3d:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0d:71:0f:3d:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:47:b7:55:ce:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 21:14:19.590207 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.590206 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 21:14:19.590321 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.590311 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 21:14:19.592059 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.592030 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 21:14:19.592232 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.592062 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-179.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 21:14:19.592975 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.592964 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 21:14:19.593036 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.592981 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 21:14:19.593036 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.593012 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:14:19.593036 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.593034 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:14:19.594015 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.594002 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 21:14:19.598012 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.597998 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 21:14:19.598166 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.598155 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 21:14:19.598914 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.598903 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 03 21:14:19.598977 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.598919 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 21:14:19.598977 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.598942 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 21:14:19.599065 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.599001 2575 kubelet.go:405] "Adding apiserver pod source" Nov 03 21:14:19.599065 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.599032 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 21:14:19.599480 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.599447 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:14:19.603182 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.603163 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 21:14:19.609312 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.609300 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:14:19.610173 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610161 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 21:14:19.610216 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610191 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 21:14:19.610364 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610354 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 21:14:19.610392 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610371 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 21:14:19.610392 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610376 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 21:14:19.610392 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610381 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 21:14:19.610392 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610387 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 21:14:19.610392 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.610392 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 21:14:19.611079 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.611070 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 21:14:19.611115 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.611080 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 21:14:19.611115 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.611087 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 21:14:19.611115 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.611092 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 21:14:19.611115 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.611100 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 21:14:19.611115 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.611108 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 21:14:19.612776 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.612764 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 21:14:19.612812 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.612780 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 21:14:19.616534 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.616521 2575 server.go:1263] "Started kubelet" Nov 03 21:14:19.616683 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.616624 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 21:14:19.616792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.616675 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 21:14:19.616835 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.616824 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 21:14:19.617316 ip-10-0-128-179 systemd[1]: Started Kubernetes Kubelet. Nov 03 21:14:19.618184 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.618149 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 21:14:19.619749 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.619733 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 03 21:14:19.629212 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.629189 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 21:14:19.629801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.629784 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 21:14:19.630353 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.630335 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 21:14:19.630485 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.630466 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:19.631110 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631081 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 21:14:19.631213 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631179 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 21:14:19.631213 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631191 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 21:14:19.631381 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631354 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 03 21:14:19.631381 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631376 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 03 21:14:19.631703 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631688 2575 factory.go:153] Registering CRI-O factory Nov 03 21:14:19.631703 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631705 2575 factory.go:223] Registration of the crio container factory successfully Nov 03 21:14:19.631801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631759 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 21:14:19.631801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631768 2575 factory.go:55] Registering systemd factory Nov 03 21:14:19.631801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631773 2575 factory.go:223] Registration of the systemd container factory successfully Nov 03 21:14:19.631801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631786 2575 factory.go:103] Registering Raw factory Nov 03 21:14:19.631801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.631798 2575 manager.go:1199] Started watching for new ooms in manager Nov 03 21:14:19.632952 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.632938 2575 manager.go:319] Starting recovery of all containers Nov 03 21:14:19.644580 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.644564 2575 manager.go:324] Recovery completed Nov 03 21:14:19.648365 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.648353 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:19.649874 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.649850 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:19.649937 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.649885 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:19.649937 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.649896 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:19.650394 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.650378 2575 cpu_manager.go:226] "Starting" policy="none" Nov 03 21:14:19.650394 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.650392 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 21:14:19.650554 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.650382 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:19.650554 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.650410 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 21:14:19.651067 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.651054 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:19.651127 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.651083 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:19.651127 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.651097 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:19.652634 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.652621 2575 policy_none.go:50] "Start" Nov 03 21:14:19.653223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.653206 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 21:14:19.653284 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.653226 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 21:14:19.655146 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.655136 2575 policy_none.go:44] "Start" Nov 03 21:14:19.692187 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.692160 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 21:14:19.694315 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.694304 2575 manager.go:341] "Starting Device Plugin manager" Nov 03 21:14:19.694958 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.694938 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 21:14:19.695011 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.694963 2575 server.go:87] "Starting device plugin registration server" Nov 03 21:14:19.695233 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.695221 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 21:14:19.695267 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.695240 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 21:14:19.695402 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.695385 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 21:14:19.695484 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.695472 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 21:14:19.695531 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.695484 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 21:14:19.695926 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.695905 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 21:14:19.695983 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.695963 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:19.755929 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.755909 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-179.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 21:14:19.757819 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.755905 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-179.us-west-1.compute.internal.18e4a571c2afdb1b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-179.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-179.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:19.616492315 +0000 UTC m=+0.349756078,LastTimestamp:2026-11-03 21:14:19.616492315 +0000 UTC m=+0.349756078,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-179.us-west-1.compute.internal,}" Nov 03 21:14:19.774252 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.774212 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 21:14:19.775560 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.775541 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 21:14:19.775630 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.775571 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 21:14:19.775630 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.775592 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 21:14:19.775715 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.775648 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 21:14:19.795544 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.795472 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:19.796379 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.796366 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:19.796433 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.796392 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:19.796433 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.796408 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:19.796433 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.796431 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.804443 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.804427 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-179.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 21:14:19.807056 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.807030 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 21:14:19.807120 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.807077 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-179.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 21:14:19.807120 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.807082 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 21:14:19.807120 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.806999 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-179.us-west-1.compute.internal.18e4a571c4ad31a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-179.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-179.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-179.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:19.649872292 +0000 UTC m=+0.383136054,LastTimestamp:2026-11-03 21:14:19.649872292 +0000 UTC m=+0.383136054,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-179.us-west-1.compute.internal,}" Nov 03 21:14:19.876586 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.876552 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal"] Nov 03 21:14:19.876688 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.876638 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:19.877513 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.877496 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:19.877556 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.877528 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:19.877556 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.877550 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:19.878069 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.878053 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 21:14:19.878105 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.878093 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kkmnr" Nov 03 21:14:19.879955 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.879943 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:19.880109 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.880146 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880134 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:19.880754 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880737 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:19.880842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880778 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:19.880842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880788 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:19.880842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880739 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:19.880930 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880845 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:19.880930 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.880867 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:19.883075 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.883059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.883163 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.883089 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:19.884308 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.884285 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:19.884381 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.884330 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:19.884381 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.884340 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:19.912305 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.912280 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.914700 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.914685 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.916393 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:19.916302 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-179.us-west-1.compute.internal.18e4a571c4ad77a9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-179.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-179.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-179.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:19.649890217 +0000 UTC m=+0.383153979,LastTimestamp:2026-11-03 21:14:19.649890217 +0000 UTC m=+0.383153979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-179.us-west-1.compute.internal,}" Nov 03 21:14:19.933055 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.933028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.933123 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.933062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1745eaf53680bc2ae777ad5a0ee0f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"1745eaf53680bc2ae777ad5a0ee0f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:19.933123 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:19.933086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.033539 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.033508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.033689 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.033545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1745eaf53680bc2ae777ad5a0ee0f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"1745eaf53680bc2ae777ad5a0ee0f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.033689 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.033562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.033689 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.033603 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.033689 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.033606 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.033689 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.033606 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1745eaf53680bc2ae777ad5a0ee0f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"1745eaf53680bc2ae777ad5a0ee0f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.076625 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.076545 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:14:20.076762 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.076748 2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-128-179.us-west-1.compute.internal\" not found" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.128967 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.128944 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-179.us-west-1.compute.internal\" not found" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.214937 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.214916 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.218064 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.218038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.277125 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.277100 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:20.277974 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.277959 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:20.278082 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.277991 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:20.278082 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.278002 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:20.278082 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.278038 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.333856 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.333809 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:20.333856 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.333832 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-179.us-west-1.compute.internal\": node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:20.447453 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.447429 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:20.547715 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.547689 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:20.561853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.561830 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 21:14:20.562008 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.561986 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 21:14:20.562053 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.561961 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a272bc74d03104c58bb06e21328e27dc-41a6df2b1383438f.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-179.us-west-1.compute.internal.18e4a571c4ad77a9\": read tcp 10.0.128.179:41868->3.130.232.210:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-179.us-west-1.compute.internal.18e4a571c4ad77a9 default 5214 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-179.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-179.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-179.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:19 +0000 UTC,LastTimestamp:2026-11-03 21:14:19.796396148 +0000 UTC m=+0.529659911,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-179.us-west-1.compute.internal,}" Nov 03 21:14:20.648309 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.648256 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:20.749032 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.749014 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:20.798843 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.798818 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:14:20.849974 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.849948 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:20.882881 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.882847 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:09:19 +0000 UTC" deadline="2028-06-10 10:09:32.53858265 +0000 UTC" Nov 03 21:14:20.882881 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.882873 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14028h55m11.655712136s" Nov 03 21:14:20.894025 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.894006 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:14:20.894999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.894982 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:14:20.895082 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.895009 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:14:20.895082 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:20.895019 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:14:20.950067 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:20.950008 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:21.050379 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.050350 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:21.150824 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.150672 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:21.156194 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.156179 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:14:21.251598 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.251531 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:14:21.251845 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.251833 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:14:21.292953 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:21.292917 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63ea962a14cc8c251237d3fd81ff3a9.slice/crio-4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c WatchSource:0}: Error finding container 4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c: Status 404 returned error can't find the container with id 4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c Nov 03 21:14:21.293146 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:21.293128 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1745eaf53680bc2ae777ad5a0ee0f5c6.slice/crio-bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722 WatchSource:0}: Error finding container bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722: Status 404 returned error can't find the container with id bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722 Nov 03 21:14:21.297004 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.296989 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:14:21.331231 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.331208 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:21.390587 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.390569 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:14:21.391486 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.391474 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:14:21.448013 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.447992 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:14:21.600371 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.600284 2575 apiserver.go:52] "Watching apiserver" Nov 03 21:14:21.630461 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.630437 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 21:14:21.697717 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.697690 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 21:14:21.697977 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.697963 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-5wwt4","kube-system/global-pull-secret-syncer-9vfbk","openshift-dns/node-resolver-rvzp6","openshift-network-operator/iptables-alerter-jqr6h","openshift-image-registry/node-ca-lrwcd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-qjndk","openshift-ovn-kubernetes/ovnkube-node-x8bjk","openshift-cluster-node-tuning-operator/tuned-lwqh9","openshift-multus/network-metrics-daemon-nkqjv","openshift-network-diagnostics/network-check-target-2xkcf","kube-system/konnectivity-agent-6ljk7","kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb"] Nov 03 21:14:21.701402 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.701388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.703285 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.703269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:21.703377 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.703362 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:21.705563 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.705551 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.707623 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.707589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.709586 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.709569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.711643 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.711628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.713775 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.713760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.716025 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.716007 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.719744 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.719727 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.720035 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.720010 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:21.722327 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.722312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:21.722410 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.722396 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:21.724481 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.724469 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.726771 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.726749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:21.732869 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.732853 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 21:14:21.743481 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-device-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.743549 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-run\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.743549 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-tmp\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.743651 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-node-log\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.743651 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-dbus\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.743651 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2j52\" (UniqueName: \"kubernetes.io/projected/10df746e-9937-45a2-ae81-c39ddc189ea6-kube-api-access-m2j52\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.743751 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7tj\" (UniqueName: \"kubernetes.io/projected/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kube-api-access-cb7tj\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.743751 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-conf\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.743799 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743759 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-tuned\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.743799 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-netd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.743852 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-socket-dir-parent\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.743852 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-netns\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.743936 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ncr4\" (UniqueName: \"kubernetes.io/projected/5a4f8473-1c87-463e-8338-ee1971bd0064-kube-api-access-5ncr4\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.743936 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-host\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.743936 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744043 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744043 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.743977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-kubelet-config\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.744043 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-k8s-cni-cncf-io\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744043 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-kubelet\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744187 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-system-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744187 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-bin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744187 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-conf-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744187 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.744187 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-etc-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744366 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-bin\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744366 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-hostroot\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744366 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.744366 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744278 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-sys-fs\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.744366 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-kubelet\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744366 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-netns\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744366 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1e57895-4080-4a05-a5b0-b859cdfd96dc-host-slash\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-var-lib-kubelet\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10df746e-9937-45a2-ae81-c39ddc189ea6-hosts-file\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-socket-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-script-lib\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-kubernetes\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1e57895-4080-4a05-a5b0-b859cdfd96dc-iptables-alerter-script\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-multus\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-daemon-config\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wln6\" (UniqueName: \"kubernetes.io/projected/e1e57895-4080-4a05-a5b0-b859cdfd96dc-kube-api-access-4wln6\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smh4k\" (UniqueName: \"kubernetes.io/projected/59fd2087-a50b-4e0a-8439-379d3094ee59-kube-api-access-smh4k\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-systemd-units\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-systemd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-ovn\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-log-socket\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-env-overrides\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744898 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-etc-kubernetes\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.744969 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02c1435f-0c51-4647-b1c2-0334a39a1af2-agent-certs\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02c1435f-0c51-4647-b1c2-0334a39a1af2-konnectivity-ca\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.744997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10df746e-9937-45a2-ae81-c39ddc189ea6-tmp-dir\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysconfig\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-lib-modules\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-var-lib-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745174 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-registration-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-slash\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.745257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-system-cni-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6f4\" (UniqueName: \"kubernetes.io/projected/7736e260-9632-4568-a43c-33b5eecd6acb-kube-api-access-rc6f4\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b136c4-f8d9-4b45-9454-ab09db908f72-host\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b136c4-f8d9-4b45-9454-ab09db908f72-serviceca\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhsz\" (UniqueName: \"kubernetes.io/projected/a49fe757-54a2-4871-b450-d48e15571c9c-kube-api-access-jrhsz\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-cnibin\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-modprobe-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-systemd\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-sys\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pfj\" (UniqueName: \"kubernetes.io/projected/01b136c4-f8d9-4b45-9454-ab09db908f72-kube-api-access-67pfj\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-config\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cad3e-0d13-4459-b970-79675d07443d-ovn-node-metrics-cert\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77j56\" (UniqueName: \"kubernetes.io/projected/4f4cad3e-0d13-4459-b970-79675d07443d-kube-api-access-77j56\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.745673 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:21.746119 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-cnibin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.746119 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-cni-binary-copy\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.746119 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.745731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-multus-certs\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.748967 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.748952 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 21:14:21.749076 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.749065 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 21:14:21.749128 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.749106 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 21:14:21.749172 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.749135 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 21:14:21.749172 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.749136 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 21:14:21.749235 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.749204 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxfhg\"" Nov 03 21:14:21.752309 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.752297 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 21:14:21.752350 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.752301 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cldzt\"" Nov 03 21:14:21.752451 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.752440 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 21:14:21.754330 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.754316 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trdz7\"" Nov 03 21:14:21.754370 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.754333 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:14:21.754399 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.754382 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 21:14:21.754433 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.754423 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 21:14:21.756500 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.756487 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 21:14:21.756500 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.756487 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 21:14:21.756634 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.756500 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 21:14:21.756634 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.756541 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4rzx\"" Nov 03 21:14:21.758507 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.758494 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 21:14:21.758507 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.758496 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzfft\"" Nov 03 21:14:21.758631 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.758509 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 21:14:21.758631 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.758545 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 21:14:21.780364 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.780325 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"1745eaf53680bc2ae777ad5a0ee0f5c6","Type":"ContainerStarted","Data":"bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722"} Nov 03 21:14:21.781231 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.781215 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"c63ea962a14cc8c251237d3fd81ff3a9","Type":"ContainerStarted","Data":"4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c"} Nov 03 21:14:21.784183 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.784162 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 21:14:21.784260 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.784193 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 21:14:21.784260 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.784207 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 21:14:21.784260 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.784216 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 21:14:21.789922 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.789911 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 21:14:21.789967 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.789954 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 21:14:21.790003 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.789968 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:14:21.794355 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.794344 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4f9ng\"" Nov 03 21:14:21.794451 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.794437 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 21:14:21.794503 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.794478 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 21:14:21.794503 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.794492 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbt6z\"" Nov 03 21:14:21.794576 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.794506 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 21:14:21.794576 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.794537 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbwk2\"" Nov 03 21:14:21.837003 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.836981 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 21:14:21.837054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.837018 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2q68h" Nov 03 21:14:21.841230 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.841219 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 21:14:21.841268 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.841255 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lzq4j\"" Nov 03 21:14:21.846404 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-netns\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.846443 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1e57895-4080-4a05-a5b0-b859cdfd96dc-host-slash\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.846443 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-var-lib-kubelet\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.846516 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10df746e-9937-45a2-ae81-c39ddc189ea6-hosts-file\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.846516 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-socket-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.846516 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-script-lib\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.846516 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-kubernetes\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.846686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1e57895-4080-4a05-a5b0-b859cdfd96dc-host-slash\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.846686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-netns\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.846686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10df746e-9937-45a2-ae81-c39ddc189ea6-hosts-file\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.846686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846533 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-var-lib-kubelet\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.846686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.846686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846630 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-kubernetes\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.846686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-socket-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.846929 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1e57895-4080-4a05-a5b0-b859cdfd96dc-iptables-alerter-script\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.846973 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.847017 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.846972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-multus\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.847017 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-daemon-config\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.847017 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.847140 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:21.847140 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wln6\" (UniqueName: \"kubernetes.io/projected/e1e57895-4080-4a05-a5b0-b859cdfd96dc-kube-api-access-4wln6\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.847140 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smh4k\" (UniqueName: \"kubernetes.io/projected/59fd2087-a50b-4e0a-8439-379d3094ee59-kube-api-access-smh4k\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.847259 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-systemd-units\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847259 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-systemd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847259 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-ovn\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847259 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-log-socket\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847428 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847390 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-script-lib\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847598 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847509 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-systemd-units\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847598 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-env-overrides\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847598 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-daemon-config\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.847598 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847546 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-multus\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.847598 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847539 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-ovn\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847598 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.847598 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847591 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-systemd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847658 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-env-overrides\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1e57895-4080-4a05-a5b0-b859cdfd96dc-iptables-alerter-script\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847710 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-etc-kubernetes\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02c1435f-0c51-4647-b1c2-0334a39a1af2-agent-certs\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02c1435f-0c51-4647-b1c2-0334a39a1af2-konnectivity-ca\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10df746e-9937-45a2-ae81-c39ddc189ea6-tmp-dir\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847847 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-etc-kubernetes\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysconfig\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.847939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-lib-modules\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-var-lib-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.847983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-registration-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848045 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848060 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-log-socket\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-slash\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysconfig\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848173 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-lib-modules\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848238 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-slash\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-system-cni-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848303 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02c1435f-0c51-4647-b1c2-0334a39a1af2-konnectivity-ca\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6f4\" (UniqueName: \"kubernetes.io/projected/7736e260-9632-4568-a43c-33b5eecd6acb-kube-api-access-rc6f4\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b136c4-f8d9-4b45-9454-ab09db908f72-host\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848373 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-registration-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.848376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b136c4-f8d9-4b45-9454-ab09db908f72-serviceca\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848414 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10df746e-9937-45a2-ae81-c39ddc189ea6-tmp-dir\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhsz\" (UniqueName: \"kubernetes.io/projected/a49fe757-54a2-4871-b450-d48e15571c9c-kube-api-access-jrhsz\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.848522 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848548 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848577 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.848599 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret podName:d8acde6b-bb34-4c16-9afe-aa4c21f59dd6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:22.348582585 +0000 UTC m=+3.081846352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret") pod "global-pull-secret-syncer-9vfbk" (UID: "d8acde6b-bb34-4c16-9afe-aa4c21f59dd6") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848628 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-var-lib-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-cnibin\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-modprobe-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-systemd\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-sys\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pfj\" (UniqueName: \"kubernetes.io/projected/01b136c4-f8d9-4b45-9454-ab09db908f72-kube-api-access-67pfj\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848766 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-config\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cad3e-0d13-4459-b970-79675d07443d-ovn-node-metrics-cert\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.849104 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848803 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b136c4-f8d9-4b45-9454-ab09db908f72-serviceca\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848810 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-system-cni-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77j56\" (UniqueName: \"kubernetes.io/projected/4f4cad3e-0d13-4459-b970-79675d07443d-kube-api-access-77j56\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848880 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-modprobe-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.848915 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-cnibin\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849080 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849099 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-config\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849350 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-systemd\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849381 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-sys\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-cnibin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-cni-binary-copy\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-multus-certs\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.849576 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.849650 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs podName:a49fe757-54a2-4871-b450-d48e15571c9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:22.349637027 +0000 UTC m=+3.082900788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs") pod "network-metrics-daemon-nkqjv" (UID: "a49fe757-54a2-4871-b450-d48e15571c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b136c4-f8d9-4b45-9454-ab09db908f72-host\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-multus-certs\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.849759 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849763 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-cnibin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-device-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-run\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-tmp\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-node-log\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-dbus\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849926 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-cni-binary-copy\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2j52\" (UniqueName: \"kubernetes.io/projected/10df746e-9937-45a2-ae81-c39ddc189ea6-kube-api-access-m2j52\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7tj\" (UniqueName: \"kubernetes.io/projected/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kube-api-access-cb7tj\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.849984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-conf\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-tuned\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-netd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-socket-dir-parent\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-netns\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncr4\" (UniqueName: \"kubernetes.io/projected/5a4f8473-1c87-463e-8338-ee1971bd0064-kube-api-access-5ncr4\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-host\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.850359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-kubelet-config\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-k8s-cni-cncf-io\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-kubelet\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-system-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-bin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-conf-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850388 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-device-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-etc-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-bin\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850455 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-run\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-hostroot\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850499 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-hostroot\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850542 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-node-log\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850642 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-dbus\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.851223 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850762 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-sys-fs\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-kubelet\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850888 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-kubelet\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850933 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-netns\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.850936 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-socket-dir-parent\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851134 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-host\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851181 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851220 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-sys-fs\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851448 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-conf\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851502 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-netd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851555 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-bin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851682 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-kubelet-config\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851723 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-system-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851726 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-kubelet\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851750 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851791 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-etc-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851841 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-bin\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.851958 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.851886 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-conf-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.852684 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.852123 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.852684 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.852356 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-k8s-cni-cncf-io\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.853031 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.853007 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cad3e-0d13-4459-b970-79675d07443d-ovn-node-metrics-cert\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.853109 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.853067 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-tmp\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.853186 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.853169 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02c1435f-0c51-4647-b1c2-0334a39a1af2-agent-certs\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:21.853575 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.853562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-tuned\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.897222 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.897198 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:14:21.897222 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.897217 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:14:21.897222 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.897226 2575 projected.go:196] Error preparing data for projected volume kube-api-access-w4fhz for pod openshift-network-diagnostics/network-check-target-2xkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:21.897376 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:21.897270 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz podName:39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de nodeName:}" failed. No retries permitted until 2026-11-03 21:14:22.397258724 +0000 UTC m=+3.130522483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w4fhz" (UniqueName: "kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz") pod "network-check-target-2xkcf" (UID: "39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:21.899151 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.899131 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smh4k\" (UniqueName: \"kubernetes.io/projected/59fd2087-a50b-4e0a-8439-379d3094ee59-kube-api-access-smh4k\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:21.900649 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.900624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhsz\" (UniqueName: \"kubernetes.io/projected/a49fe757-54a2-4871-b450-d48e15571c9c-kube-api-access-jrhsz\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:21.900921 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.900895 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6f4\" (UniqueName: \"kubernetes.io/projected/7736e260-9632-4568-a43c-33b5eecd6acb-kube-api-access-rc6f4\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:21.901013 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.900999 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pfj\" (UniqueName: \"kubernetes.io/projected/01b136c4-f8d9-4b45-9454-ab09db908f72-kube-api-access-67pfj\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:21.901334 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.901310 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wln6\" (UniqueName: \"kubernetes.io/projected/e1e57895-4080-4a05-a5b0-b859cdfd96dc-kube-api-access-4wln6\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:21.902272 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.902250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncr4\" (UniqueName: \"kubernetes.io/projected/5a4f8473-1c87-463e-8338-ee1971bd0064-kube-api-access-5ncr4\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:14:21.902353 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.902327 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7tj\" (UniqueName: \"kubernetes.io/projected/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kube-api-access-cb7tj\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:21.902538 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.902516 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77j56\" (UniqueName: \"kubernetes.io/projected/4f4cad3e-0d13-4459-b970-79675d07443d-kube-api-access-77j56\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:21.902736 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:21.902720 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2j52\" (UniqueName: \"kubernetes.io/projected/10df746e-9937-45a2-ae81-c39ddc189ea6-kube-api-access-m2j52\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:22.022359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.022336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:14:22.025489 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.025463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:14:22.029074 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.029061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:14:22.030107 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.030096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:14:22.034485 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.034467 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10df746e_9937_45a2_ae81_c39ddc189ea6.slice/crio-94dc94fee5363e04e224b6fe3ba7458e32191b1d97e69cbcda60ed9a6df736a6 WatchSource:0}: Error finding container 94dc94fee5363e04e224b6fe3ba7458e32191b1d97e69cbcda60ed9a6df736a6: Status 404 returned error can't find the container with id 94dc94fee5363e04e224b6fe3ba7458e32191b1d97e69cbcda60ed9a6df736a6 Nov 03 21:14:22.037841 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.037821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:14:22.038748 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.038718 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b136c4_f8d9_4b45_9454_ab09db908f72.slice/crio-d0b901aef19e9ae7ac32978455430ac3537e8c68317adfd6daefe5b0cf5c495b WatchSource:0}: Error finding container d0b901aef19e9ae7ac32978455430ac3537e8c68317adfd6daefe5b0cf5c495b: Status 404 returned error can't find the container with id d0b901aef19e9ae7ac32978455430ac3537e8c68317adfd6daefe5b0cf5c495b Nov 03 21:14:22.039903 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.039884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:14:22.041152 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.041133 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e57895_4080_4a05_a5b0_b859cdfd96dc.slice/crio-153b1abd7c935b1a467680a61676c9e0d4c288e22d517c904a18cea07a2edacf WatchSource:0}: Error finding container 153b1abd7c935b1a467680a61676c9e0d4c288e22d517c904a18cea07a2edacf: Status 404 returned error can't find the container with id 153b1abd7c935b1a467680a61676c9e0d4c288e22d517c904a18cea07a2edacf Nov 03 21:14:22.046760 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.046735 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode346f0cc_7ed6_42f5_a4f9_1c16320cefd9.slice/crio-51c80b41da04d7ba26386c909d3b07e93a25cc458b0d82ffc9cde5d8efa04452 WatchSource:0}: Error finding container 51c80b41da04d7ba26386c909d3b07e93a25cc458b0d82ffc9cde5d8efa04452: Status 404 returned error can't find the container with id 51c80b41da04d7ba26386c909d3b07e93a25cc458b0d82ffc9cde5d8efa04452 Nov 03 21:14:22.050521 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.050500 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fd2087_a50b_4e0a_8439_379d3094ee59.slice/crio-aeec5b1108c0661a42b46d9f9eba568d9c104765f0e7e8ba9a31999801dd0011 WatchSource:0}: Error finding container aeec5b1108c0661a42b46d9f9eba568d9c104765f0e7e8ba9a31999801dd0011: Status 404 returned error can't find the container with id aeec5b1108c0661a42b46d9f9eba568d9c104765f0e7e8ba9a31999801dd0011 Nov 03 21:14:22.053541 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.053521 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5wwt4" Nov 03 21:14:22.056236 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.056222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:22.061521 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.061502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:22.061720 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.061702 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4f8473_1c87_463e_8338_ee1971bd0064.slice/crio-653ad286691eb3a87afecd5d82fb57fb931e7c0087a3e07358fc4d91241450c5 WatchSource:0}: Error finding container 653ad286691eb3a87afecd5d82fb57fb931e7c0087a3e07358fc4d91241450c5: Status 404 returned error can't find the container with id 653ad286691eb3a87afecd5d82fb57fb931e7c0087a3e07358fc4d91241450c5 Nov 03 21:14:22.064222 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.064201 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c1435f_0c51_4647_b1c2_0334a39a1af2.slice/crio-1f0f1d254373a18301419cbd223a7bc66a5878b43e4cd615e1bee8a51f8644be WatchSource:0}: Error finding container 1f0f1d254373a18301419cbd223a7bc66a5878b43e4cd615e1bee8a51f8644be: Status 404 returned error can't find the container with id 1f0f1d254373a18301419cbd223a7bc66a5878b43e4cd615e1bee8a51f8644be Nov 03 21:14:22.064693 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.064674 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:14:22.069644 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:14:22.069628 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4cad3e_0d13_4459_b970_79675d07443d.slice/crio-38c585b403cc8fb42f8498fa7259ece6fb9a1137be5069db08c526f58f27ade8 WatchSource:0}: Error finding container 38c585b403cc8fb42f8498fa7259ece6fb9a1137be5069db08c526f58f27ade8: Status 404 returned error can't find the container with id 38c585b403cc8fb42f8498fa7259ece6fb9a1137be5069db08c526f58f27ade8 Nov 03 21:14:22.354416 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.354388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:22.354553 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.354437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:22.354590 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.354571 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:22.354663 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.354634 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs podName:a49fe757-54a2-4871-b450-d48e15571c9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:23.354622431 +0000 UTC m=+4.087886191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs") pod "network-metrics-daemon-nkqjv" (UID: "a49fe757-54a2-4871-b450-d48e15571c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:22.354708 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.354675 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:22.354708 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.354698 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret podName:d8acde6b-bb34-4c16-9afe-aa4c21f59dd6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:23.354691016 +0000 UTC m=+4.087954764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret") pod "global-pull-secret-syncer-9vfbk" (UID: "d8acde6b-bb34-4c16-9afe-aa4c21f59dd6") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:22.455391 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.455356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:22.455569 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.455520 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:14:22.455569 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.455541 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:14:22.455569 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.455551 2575 projected.go:196] Error preparing data for projected volume kube-api-access-w4fhz for pod openshift-network-diagnostics/network-check-target-2xkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:22.455710 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:22.455628 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz podName:39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de nodeName:}" failed. No retries permitted until 2026-11-03 21:14:23.455588254 +0000 UTC m=+4.188852018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4fhz" (UniqueName: "kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz") pod "network-check-target-2xkcf" (UID: "39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:22.784749 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.784677 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"38c585b403cc8fb42f8498fa7259ece6fb9a1137be5069db08c526f58f27ade8"} Nov 03 21:14:22.786604 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.786567 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ljk7" event={"ID":"02c1435f-0c51-4647-b1c2-0334a39a1af2","Type":"ContainerStarted","Data":"1f0f1d254373a18301419cbd223a7bc66a5878b43e4cd615e1bee8a51f8644be"} Nov 03 21:14:22.787805 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.787759 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" event={"ID":"59fd2087-a50b-4e0a-8439-379d3094ee59","Type":"ContainerStarted","Data":"aeec5b1108c0661a42b46d9f9eba568d9c104765f0e7e8ba9a31999801dd0011"} Nov 03 21:14:22.788926 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.788900 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqr6h" event={"ID":"e1e57895-4080-4a05-a5b0-b859cdfd96dc","Type":"ContainerStarted","Data":"153b1abd7c935b1a467680a61676c9e0d4c288e22d517c904a18cea07a2edacf"} Nov 03 21:14:22.790012 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.789988 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrwcd" event={"ID":"01b136c4-f8d9-4b45-9454-ab09db908f72","Type":"ContainerStarted","Data":"d0b901aef19e9ae7ac32978455430ac3537e8c68317adfd6daefe5b0cf5c495b"} Nov 03 21:14:22.791433 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.791412 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerStarted","Data":"ac1476918d6cf0940208a74aea0cbef9585642173751d9e7adbdf93d52fd02a1"} Nov 03 21:14:22.792890 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.792869 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvzp6" event={"ID":"10df746e-9937-45a2-ae81-c39ddc189ea6","Type":"ContainerStarted","Data":"94dc94fee5363e04e224b6fe3ba7458e32191b1d97e69cbcda60ed9a6df736a6"} Nov 03 21:14:22.794156 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.794129 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5wwt4" event={"ID":"5a4f8473-1c87-463e-8338-ee1971bd0064","Type":"ContainerStarted","Data":"653ad286691eb3a87afecd5d82fb57fb931e7c0087a3e07358fc4d91241450c5"} Nov 03 21:14:22.795352 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.795332 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" event={"ID":"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9","Type":"ContainerStarted","Data":"51c80b41da04d7ba26386c909d3b07e93a25cc458b0d82ffc9cde5d8efa04452"} Nov 03 21:14:22.839859 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.839828 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:21 +0000 UTC" deadline="2028-06-19 00:13:20.892530991 +0000 UTC" Nov 03 21:14:22.839994 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:22.839871 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14234h58m58.052673504s" Nov 03 21:14:23.362680 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.362593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:23.362680 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.362661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:23.362869 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.362778 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:23.362869 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.362834 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs podName:a49fe757-54a2-4871-b450-d48e15571c9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:25.362815987 +0000 UTC m=+6.096079758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs") pod "network-metrics-daemon-nkqjv" (UID: "a49fe757-54a2-4871-b450-d48e15571c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:23.362976 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.362893 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:23.362976 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.362926 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret podName:d8acde6b-bb34-4c16-9afe-aa4c21f59dd6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:25.362916489 +0000 UTC m=+6.096180243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret") pod "global-pull-secret-syncer-9vfbk" (UID: "d8acde6b-bb34-4c16-9afe-aa4c21f59dd6") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:23.463828 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.463784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:23.464015 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.463995 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:14:23.464088 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.464025 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:14:23.464088 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.464037 2575 projected.go:196] Error preparing data for projected volume kube-api-access-w4fhz for pod openshift-network-diagnostics/network-check-target-2xkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:23.464186 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.464095 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz podName:39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de nodeName:}" failed. No retries permitted until 2026-11-03 21:14:25.464076909 +0000 UTC m=+6.197340672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4fhz" (UniqueName: "kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz") pod "network-check-target-2xkcf" (UID: "39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:23.778653 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.778556 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:23.778800 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.778712 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:23.778858 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.778842 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:23.778995 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.778972 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:23.779069 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.779047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:23.779153 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:23.779134 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:23.840487 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.840425 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:21 +0000 UTC" deadline="2028-08-14 12:51:04.926903406 +0000 UTC" Nov 03 21:14:23.840487 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:23.840485 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15591h36m41.086422564s" Nov 03 21:14:24.813374 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:24.813097 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"1745eaf53680bc2ae777ad5a0ee0f5c6","Type":"ContainerStarted","Data":"3850cf730c19090b6dda10f6a664ac307d54fcf6d556fb578329272a5ab0cf7f"} Nov 03 21:14:24.817113 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:24.817080 2575 generic.go:356] "Generic (PLEG): container finished" podID="c63ea962a14cc8c251237d3fd81ff3a9" containerID="ccf5087107f5f7564802d8e275656cebe9971b2f2edf267e135a619b2e14623d" exitCode=0 Nov 03 21:14:24.817238 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:24.817123 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"c63ea962a14cc8c251237d3fd81ff3a9","Type":"ContainerDied","Data":"ccf5087107f5f7564802d8e275656cebe9971b2f2edf267e135a619b2e14623d"} Nov 03 21:14:25.088436 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.087634 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" podStartSLOduration=4.087597033 podStartE2EDuration="4.087597033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:24.97546175 +0000 UTC m=+5.708725523" watchObservedRunningTime="2026-11-03 21:14:25.087597033 +0000 UTC m=+5.820860808" Nov 03 21:14:25.383992 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.383950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:25.384141 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.384000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:25.384196 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.384174 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:25.384245 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.384239 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs podName:a49fe757-54a2-4871-b450-d48e15571c9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:29.384218987 +0000 UTC m=+10.117482750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs") pod "network-metrics-daemon-nkqjv" (UID: "a49fe757-54a2-4871-b450-d48e15571c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:25.384377 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.384361 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:25.384452 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.384410 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret podName:d8acde6b-bb34-4c16-9afe-aa4c21f59dd6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:29.38439652 +0000 UTC m=+10.117660292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret") pod "global-pull-secret-syncer-9vfbk" (UID: "d8acde6b-bb34-4c16-9afe-aa4c21f59dd6") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:25.485549 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.485224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:25.485549 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.485425 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:14:25.485549 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.485442 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:14:25.485549 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.485454 2575 projected.go:196] Error preparing data for projected volume kube-api-access-w4fhz for pod openshift-network-diagnostics/network-check-target-2xkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:25.485549 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.485513 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz podName:39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de nodeName:}" failed. No retries permitted until 2026-11-03 21:14:29.485497175 +0000 UTC m=+10.218760942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4fhz" (UniqueName: "kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz") pod "network-check-target-2xkcf" (UID: "39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:25.776106 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.776025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:25.776257 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.776166 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:25.776257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.776222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:25.776357 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.776288 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:25.776357 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.776327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:25.776454 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:25.776388 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:25.825101 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.825061 2575 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="13cd182d80f098e3652b88beefd16144ecec709d3f6a889f86a64192816c7983" exitCode=0 Nov 03 21:14:25.825256 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.825165 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerDied","Data":"13cd182d80f098e3652b88beefd16144ecec709d3f6a889f86a64192816c7983"} Nov 03 21:14:25.838356 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:25.838327 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"c63ea962a14cc8c251237d3fd81ff3a9","Type":"ContainerStarted","Data":"c66f79e2b1f12490f94dd6acfc695247002abe83e4ba13307f916c4a9e1b0e9c"} Nov 03 21:14:26.847513 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:26.847475 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ljk7" event={"ID":"02c1435f-0c51-4647-b1c2-0334a39a1af2","Type":"ContainerStarted","Data":"fb62d0943a5d9fc4a09370700429c32ab75bb357ddabcceb659fa127f5b35f2c"} Nov 03 21:14:26.959388 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:26.958499 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" podStartSLOduration=5.958477182 podStartE2EDuration="5.958477182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:26.102199511 +0000 UTC m=+6.835463285" watchObservedRunningTime="2026-11-03 21:14:26.958477182 +0000 UTC m=+7.691740955" Nov 03 21:14:27.780690 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:27.780662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:27.780850 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:27.780790 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:27.780925 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:27.780860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:27.780979 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:27.780935 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:27.781038 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:27.780995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:27.781088 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:27.781068 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:27.860731 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:27.860685 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrwcd" event={"ID":"01b136c4-f8d9-4b45-9454-ab09db908f72","Type":"ContainerStarted","Data":"aaba53c5c33305de9c64e0a947b0be0f014bc6b32f33781d138008c24cea4d43"} Nov 03 21:14:27.973529 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:27.973472 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6ljk7" podStartSLOduration=3.869832515 podStartE2EDuration="7.973452277s" totalImagesPullingTime="4.103619762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:26.959356521 +0000 UTC m=+7.692620294" watchObservedRunningTime="2026-11-03 21:14:27.973452277 +0000 UTC m=+8.706716050" Nov 03 21:14:28.552160 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:28.552124 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:28.553019 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:28.552963 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:28.665642 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:28.665568 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lrwcd" podStartSLOduration=3.448764077 podStartE2EDuration="8.665548108s" totalImagesPullingTime="5.216784031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:27.974212656 +0000 UTC m=+8.707476447" watchObservedRunningTime="2026-11-03 21:14:28.665548108 +0000 UTC m=+9.398811881" Nov 03 21:14:28.868434 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:28.868403 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" event={"ID":"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9","Type":"ContainerStarted","Data":"6c9dc233d8d27c73c26606ff1037d7a8311c2ad8affbd52de443a166b810ad37"} Nov 03 21:14:29.416162 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.415870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:29.416162 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.415909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:29.416162 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.416007 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:29.416162 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.416053 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs podName:a49fe757-54a2-4871-b450-d48e15571c9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.416042019 +0000 UTC m=+18.149305770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs") pod "network-metrics-daemon-nkqjv" (UID: "a49fe757-54a2-4871-b450-d48e15571c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:29.416162 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.416090 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:29.416162 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.416109 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret podName:d8acde6b-bb34-4c16-9afe-aa4c21f59dd6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.41610363 +0000 UTC m=+18.149367380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret") pod "global-pull-secret-syncer-9vfbk" (UID: "d8acde6b-bb34-4c16-9afe-aa4c21f59dd6") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:29.516700 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.516434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:29.516700 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.516579 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:14:29.516700 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.516592 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:14:29.516700 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.516601 2575 projected.go:196] Error preparing data for projected volume kube-api-access-w4fhz for pod openshift-network-diagnostics/network-check-target-2xkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:29.516700 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.516664 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz podName:39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de nodeName:}" failed. No retries permitted until 2026-11-03 21:14:37.516651909 +0000 UTC m=+18.249915659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4fhz" (UniqueName: "kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz") pod "network-check-target-2xkcf" (UID: "39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:29.781361 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.780927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:29.781361 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.781040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:29.781361 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.781116 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:29.781361 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.781195 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:29.781361 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.781250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:29.781361 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:29.781327 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:29.877560 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.877529 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:14:29.878686 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.878446 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5wwt4" event={"ID":"5a4f8473-1c87-463e-8338-ee1971bd0064","Type":"ContainerStarted","Data":"f7f810a2f6330468c481192c1ddc065bdbec193a8cf42a59920ade51c95491b3"} Nov 03 21:14:29.967527 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.967480 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 21:14:29.992768 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:29.992487 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5wwt4" podStartSLOduration=2.358109485 podStartE2EDuration="9.992473508s" totalImagesPullingTime="7.634364023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:29.991586733 +0000 UTC m=+10.724850505" watchObservedRunningTime="2026-11-03 21:14:29.992473508 +0000 UTC m=+10.725737278" Nov 03 21:14:30.701151 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:30.701015 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:14:29.967496696Z","UUID":"4cee4b82-e884-41bf-bf63-360062579a65","Handler":null,"Name":"","Endpoint":""} Nov 03 21:14:30.702660 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:30.702638 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 21:14:30.702776 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:30.702667 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 21:14:30.883805 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:30.883767 2575 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="b36323632826442ef943c0257a6ed2e953ba5a9d7d5461471eee7e851448ccda" exitCode=0 Nov 03 21:14:30.884376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:30.883850 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerDied","Data":"b36323632826442ef943c0257a6ed2e953ba5a9d7d5461471eee7e851448ccda"} Nov 03 21:14:30.886401 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:30.886348 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvzp6" event={"ID":"10df746e-9937-45a2-ae81-c39ddc189ea6","Type":"ContainerStarted","Data":"ae8114a926faba953314efc492dc7eb2fdb8356d7d92cd57825f45ce1dd7bcff"} Nov 03 21:14:30.890332 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:30.890305 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" event={"ID":"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9","Type":"ContainerStarted","Data":"dd3b1301ca283456f5a5b88100b41a54bbb3a83d119ff902ec2dee378ac15392"} Nov 03 21:14:31.116002 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:31.115936 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rvzp6" podStartSLOduration=2.991324082 podStartE2EDuration="11.115917301s" totalImagesPullingTime="8.124593219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:31.115826601 +0000 UTC m=+11.849090372" watchObservedRunningTime="2026-11-03 21:14:31.115917301 +0000 UTC m=+11.849181072" Nov 03 21:14:31.778801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:31.776700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:31.778801 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:31.776832 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:31.778801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:31.776900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:31.778801 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:31.776947 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:31.778801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:31.776981 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:31.778801 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:31.777040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:31.893967 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:31.893931 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqr6h" event={"ID":"e1e57895-4080-4a05-a5b0-b859cdfd96dc","Type":"ContainerStarted","Data":"577d828e09edd6e52f274d839d6fb0b8c96f39acd615f24bba1ba4cf99cdb180"} Nov 03 21:14:31.896570 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:31.896541 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" event={"ID":"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9","Type":"ContainerStarted","Data":"2712ced29cc149e1a9761188b5c170de2eda380ddefcde223256187ebfdeab0d"} Nov 03 21:14:32.005054 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:32.004834 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jqr6h" podStartSLOduration=3.782609109 podStartE2EDuration="12.00481454s" totalImagesPullingTime="8.222205431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:32.004237907 +0000 UTC m=+12.737501681" watchObservedRunningTime="2026-11-03 21:14:32.00481454 +0000 UTC m=+12.738078312" Nov 03 21:14:32.117887 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:32.117828 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" podStartSLOduration=3.198276744 podStartE2EDuration="12.117808947s" totalImagesPullingTime="8.919532203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:32.117167376 +0000 UTC m=+12.850431147" watchObservedRunningTime="2026-11-03 21:14:32.117808947 +0000 UTC m=+12.851072720" Nov 03 21:14:32.900778 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:32.900738 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" event={"ID":"59fd2087-a50b-4e0a-8439-379d3094ee59","Type":"ContainerStarted","Data":"977da2a30d9d9ddf5e2f0ffa18256a0c923342690ee30604f80ea3f4a05cf48d"} Nov 03 21:14:32.903359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:32.903247 2575 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="39a23047b5528a36aab4da63d5642cf013dd320b85024f9c3a97a0c0ee327f78" exitCode=0 Nov 03 21:14:32.903359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:32.903296 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerDied","Data":"39a23047b5528a36aab4da63d5642cf013dd320b85024f9c3a97a0c0ee327f78"} Nov 03 21:14:33.013966 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:33.013913 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" podStartSLOduration=3.072858093 podStartE2EDuration="13.013900387s" totalImagesPullingTime="9.941042294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:33.013602481 +0000 UTC m=+13.746866254" watchObservedRunningTime="2026-11-03 21:14:33.013900387 +0000 UTC m=+13.747164158" Nov 03 21:14:33.775827 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:33.775795 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:33.775827 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:33.775813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:33.776074 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:33.775886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:33.776074 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:33.775989 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:33.776074 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:33.776067 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:33.776196 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:33.776145 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:34.821164 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:34.821128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:34.909821 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:34.909787 2575 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="093bb622a293fda525d7ada3078750df1cff6b222b18cfc09c4d7692741511cf" exitCode=0 Nov 03 21:14:34.909976 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:34.909839 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerDied","Data":"093bb622a293fda525d7ada3078750df1cff6b222b18cfc09c4d7692741511cf"} Nov 03 21:14:35.100880 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:35.100844 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:35.776692 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:35.776662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:35.776862 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:35.776733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:35.776862 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:35.776730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:35.776982 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:35.776850 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:35.777072 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:35.776981 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:35.777072 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:35.777053 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:36.917461 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.917305 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:14:36.917924 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.917796 2575 generic.go:356] "Generic (PLEG): container finished" podID="4f4cad3e-0d13-4459-b970-79675d07443d" containerID="7107698647ae1637a61db56dd070e6dd04eccc291eb636d6ebb7ad4e99392dfd" exitCode=1 Nov 03 21:14:36.917924 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.917849 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"052520fa5508d1c0ed62ef2a2d0110cd3e168758a76a000431c4b85dee91b232"} Nov 03 21:14:36.917924 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.917903 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"b99703f16944b4d9d262cc715cdd1b0364f5b1ea9096c33ae024b95283ba0929"} Nov 03 21:14:36.918008 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.917932 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"85e290b224b1eb08521e9ebf5c73173a3387080454a1cc96e8198b643c72f850"} Nov 03 21:14:36.918008 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.917958 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"9a3c1656f5bb7c64e1693c1aaaae561fee9417b003dadbdf3b02a16f2a6bc617"} Nov 03 21:14:36.918008 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.917983 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerDied","Data":"7107698647ae1637a61db56dd070e6dd04eccc291eb636d6ebb7ad4e99392dfd"} Nov 03 21:14:36.918095 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:36.918009 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"e9e050d5fe00fabecf3f5f56ff4118eb66c2b8a91ada157330ba76d34439b761"} Nov 03 21:14:37.477631 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.477414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:37.477814 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.477654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:37.477814 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.477555 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:37.477814 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.477760 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret podName:d8acde6b-bb34-4c16-9afe-aa4c21f59dd6 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:53.477742591 +0000 UTC m=+34.211006361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret") pod "global-pull-secret-syncer-9vfbk" (UID: "d8acde6b-bb34-4c16-9afe-aa4c21f59dd6") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:37.477814 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.477788 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:37.477941 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.477827 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs podName:a49fe757-54a2-4871-b450-d48e15571c9c nodeName:}" failed. No retries permitted until 2026-11-03 21:14:53.477815597 +0000 UTC m=+34.211079371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs") pod "network-metrics-daemon-nkqjv" (UID: "a49fe757-54a2-4871-b450-d48e15571c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:37.578869 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.578844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:37.579024 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.578955 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:14:37.579024 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.578969 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:14:37.579024 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.578977 2575 projected.go:196] Error preparing data for projected volume kube-api-access-w4fhz for pod openshift-network-diagnostics/network-check-target-2xkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:37.579024 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.579018 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz podName:39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de nodeName:}" failed. No retries permitted until 2026-11-03 21:14:53.579005802 +0000 UTC m=+34.312269566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4fhz" (UniqueName: "kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz") pod "network-check-target-2xkcf" (UID: "39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:37.736726 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.736640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:37.775836 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.775812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:37.775955 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.775851 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:37.775955 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.775932 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:37.776029 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.775992 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:37.776029 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.776020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:37.776125 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:37.776109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:37.922316 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.922285 2575 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="afdf65282ae5675c5035d67cd1ebddf00f398fd6535aa34033961033b34fac6c" exitCode=0 Nov 03 21:14:37.922708 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.922330 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerDied","Data":"afdf65282ae5675c5035d67cd1ebddf00f398fd6535aa34033961033b34fac6c"} Nov 03 21:14:37.963157 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:37.963131 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:38.734066 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:38.734032 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:38.926860 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:38.926828 2575 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="31de5d3a0376d4b66f3510706996a37aa356a4bd858f5f4346df57db03aa5a96" exitCode=0 Nov 03 21:14:38.927332 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:38.926890 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerDied","Data":"31de5d3a0376d4b66f3510706996a37aa356a4bd858f5f4346df57db03aa5a96"} Nov 03 21:14:38.929939 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:38.929919 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:14:38.930295 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:38.930276 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"a31e55e38c96b7fc5005be2d1b5e556c9cf5f84250cf58beefc4ae39c6d4fc3e"} Nov 03 21:14:39.776585 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:39.776557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:39.776792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:39.776679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:39.776792 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:39.776690 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:39.776792 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:39.776750 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:39.776792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:39.776775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:39.777014 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:39.776834 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:39.935648 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:39.935603 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjndk" event={"ID":"7736e260-9632-4568-a43c-33b5eecd6acb","Type":"ContainerStarted","Data":"b837f2a7ec20c2f76a669c41a8b50c4f3d351188ca17b9cdf2c53a4a4a524849"} Nov 03 21:14:40.054947 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:40.054856 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qjndk" podStartSLOduration=8.215305315 podStartE2EDuration="20.05484403s" totalImagesPullingTime="11.839538715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:40.054319326 +0000 UTC m=+20.787583099" watchObservedRunningTime="2026-11-03 21:14:40.05484403 +0000 UTC m=+20.788107800" Nov 03 21:14:40.940756 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:40.940677 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:14:40.941515 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:40.941020 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"76ca57fc63302d0b5d27797e5026dd1abd61e2ea9e4bedbb053ecfcc53cccee5"} Nov 03 21:14:40.941681 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:40.941666 2575 scope.go:122] "RemoveContainer" containerID="7107698647ae1637a61db56dd070e6dd04eccc291eb636d6ebb7ad4e99392dfd" Nov 03 21:14:41.061663 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.061636 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:41.286682 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.286579 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:41.683592 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.683559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:41.776178 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.776141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:41.776178 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.776164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:41.776376 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:41.776257 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:41.776376 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:41.776290 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:41.776376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.776344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:41.776499 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:41.776402 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:41.919652 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.919458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:41.946726 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.946664 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:14:41.947131 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.947006 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" event={"ID":"4f4cad3e-0d13-4459-b970-79675d07443d","Type":"ContainerStarted","Data":"8dcbf84068a9142adb54123c757beee5b62eaec78e4e89e29acca6fea63b54a8"} Nov 03 21:14:41.947346 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.947327 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:41.947486 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.947451 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:41.947544 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.947488 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:41.961779 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.961755 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:41.961896 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:41.961849 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:14:42.189452 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:42.189394 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" podStartSLOduration=8.220447215 podStartE2EDuration="22.189383279s" totalImagesPullingTime="13.968936064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:42.15826822 +0000 UTC m=+22.891531993" watchObservedRunningTime="2026-11-03 21:14:42.189383279 +0000 UTC m=+22.922647050" Nov 03 21:14:42.273663 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:42.273563 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:42.512649 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:42.512601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:42.742582 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:42.742552 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:43.373603 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:43.373569 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9vfbk"] Nov 03 21:14:43.374176 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:43.373750 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:43.374176 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:43.373854 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:43.374265 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:43.374249 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2xkcf"] Nov 03 21:14:43.374349 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:43.374337 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:43.374427 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:43.374411 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:43.376234 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:43.376211 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkqjv"] Nov 03 21:14:43.376346 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:43.376319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:43.376448 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:43.376424 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:43.795593 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:43.795511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:44.060149 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:44.060072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:44.649504 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:44.649471 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:44.776672 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:44.776644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:44.776807 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:44.776649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:44.776807 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:44.776750 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:44.776881 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:44.776652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:44.776881 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:44.776820 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:44.776949 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:44.776894 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:45.724094 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:45.724057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:46.146101 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:46.146068 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:46.397800 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:46.397731 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:46.617806 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:46.617776 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:46.776324 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:46.776235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:46.776324 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:46.776291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:46.776773 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:46.776242 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:46.776773 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:46.776374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:46.776773 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:46.776442 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:46.776773 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:46.776522 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:46.839002 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:46.838978 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:47.135198 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:47.135167 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:47.613734 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:47.613706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:47.641972 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:47.641951 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rvzp6_10df746e-9937-45a2-ae81-c39ddc189ea6/dns-node-resolver/0.log" Nov 03 21:14:47.842625 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:47.842573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:48.064055 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:48.063972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:48.172575 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:48.172549 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrwcd_01b136c4-f8d9-4b45-9454-ab09db908f72/node-ca/0.log" Nov 03 21:14:48.285487 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:48.285461 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:48.505116 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:48.505085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:48.776720 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:48.776651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:48.776853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:48.776743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:48.776853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:48.776753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:48.776853 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:48.776750 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:48.776853 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:48.776825 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:48.776976 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:48.776889 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:49.386389 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:49.386352 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:49.427172 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:49.427146 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:49.427317 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:49.427279 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:14:49.427791 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:49.427772 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:14:49.626234 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:49.626204 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:49.855811 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:49.855780 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:50.080784 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:50.080748 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:50.300488 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:50.300410 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:50.529932 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:50.529899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:50.776010 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:50.775980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:50.776010 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:50.776011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:50.776193 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:50.776082 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:50.776193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:50.776118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:50.776259 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:50.776203 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:50.776290 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:50.776259 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:51.339845 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:51.339813 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:51.608661 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:51.608570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:51.830261 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:51.830221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:52.055088 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:52.055003 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:52.279512 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:52.279474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:52.502242 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:52.502208 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:52.776763 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:52.776690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:52.776763 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:52.776724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:52.777236 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:52.776809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:52.777236 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:52.776806 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:52.777236 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:52.776862 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:52.777236 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:52.776950 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:53.496195 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:53.496156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:53.496195 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:53.496200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:53.496434 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.496299 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:53.496434 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.496304 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:53.496434 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.496356 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret podName:d8acde6b-bb34-4c16-9afe-aa4c21f59dd6 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:25.496341356 +0000 UTC m=+66.229605107 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret") pod "global-pull-secret-syncer-9vfbk" (UID: "d8acde6b-bb34-4c16-9afe-aa4c21f59dd6") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:14:53.496434 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.496370 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs podName:a49fe757-54a2-4871-b450-d48e15571c9c nodeName:}" failed. No retries permitted until 2026-11-03 21:15:25.49636405 +0000 UTC m=+66.229627800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs") pod "network-metrics-daemon-nkqjv" (UID: "a49fe757-54a2-4871-b450-d48e15571c9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:14:53.597564 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:53.597528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:53.597717 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.597702 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:14:53.597755 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.597723 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:14:53.597755 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.597732 2575 projected.go:196] Error preparing data for projected volume kube-api-access-w4fhz for pod openshift-network-diagnostics/network-check-target-2xkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:53.597819 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:53.597779 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz podName:39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de nodeName:}" failed. No retries permitted until 2026-11-03 21:15:25.597767134 +0000 UTC m=+66.331030888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4fhz" (UniqueName: "kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz") pod "network-check-target-2xkcf" (UID: "39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:14:53.609772 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:53.609738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:53.833717 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:53.833644 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:54.060757 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:54.060724 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:54.288843 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:54.288811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:54.522457 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:54.522429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:54.775772 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:54.775742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:54.775945 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:54.775835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:54.775945 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:54.775842 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:54.775945 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:54.775859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:54.775945 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:54.775926 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:54.776081 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:54.776001 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:54.933964 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:54.933933 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:55.198240 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:55.198211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:55.431513 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:55.431483 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:55.670271 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:55.670237 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:55.892646 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:55.892597 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:56.134296 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:56.134268 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:56.371338 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:56.371312 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:56.605509 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:56.605486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:56.776102 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:56.776072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:56.776102 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:56.776089 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:56.776347 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:56.776072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:56.776347 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:56.776182 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:14:56.776347 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:56.776265 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:56.776453 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:56.776349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:56.832531 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:56.832497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:58.775801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:58.775769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:14:58.775801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:58.775793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:14:58.775801 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:14:58.775812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:14:58.776342 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:58.775898 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:14:58.776342 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:58.775947 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:14:58.776342 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:14:58.776017 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:15:00.522548 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:00.522513 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:00.748568 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:00.748538 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:00.776558 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:00.776487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:00.776558 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:00.776530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:00.776763 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:00.776607 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:15:00.776763 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:00.776666 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:15:00.776763 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:00.776690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:00.776916 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:00.776770 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:15:00.979359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:00.979328 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:01.208871 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:01.208838 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:01.468496 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:01.468414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:02.776127 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:02.776084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:02.776127 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:02.776132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:02.776544 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:02.776192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:02.776544 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:02.776208 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:15:02.776544 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:02.776256 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:15:02.776544 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:02.776342 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:15:02.929250 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:02.929209 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:03.149803 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:03.149774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:04.447210 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:04.447172 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:04.673671 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:04.673640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:04.776248 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:04.776156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:04.776248 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:04.776190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:04.776248 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:04.776197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:04.776486 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:04.776305 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9vfbk" podUID="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" Nov 03 21:15:04.776486 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:04.776367 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjv" podUID="a49fe757-54a2-4871-b450-d48e15571c9c" Nov 03 21:15:04.776486 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:04.776423 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xkcf" podUID="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" Nov 03 21:15:05.104023 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.103990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:05.332129 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.332100 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:05.526257 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.526231 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeReady" Nov 03 21:15:05.526591 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.526340 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 21:15:05.687215 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.687142 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9srkw"] Nov 03 21:15:05.688812 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.688797 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thpr2"] Nov 03 21:15:05.688934 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.688915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.690136 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.690118 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xcf7l"] Nov 03 21:15:05.690249 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.690233 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:05.691669 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.691655 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9srkw"] Nov 03 21:15:05.691743 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.691721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.708150 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.708133 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thpr2"] Nov 03 21:15:05.711269 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.711248 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xcf7l"] Nov 03 21:15:05.740977 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.740954 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 21:15:05.741083 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.741065 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8l7\"" Nov 03 21:15:05.741083 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.741078 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 21:15:05.746486 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.746471 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 21:15:05.746554 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.746543 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rf5\"" Nov 03 21:15:05.746587 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.746557 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 21:15:05.789152 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.789135 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tq7pf\"" Nov 03 21:15:05.789152 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.789139 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 21:15:05.791388 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7xd\" (UniqueName: \"kubernetes.io/projected/b10257bd-92be-4617-beab-f2270c5d396d-kube-api-access-bc7xd\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.791455 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d5c0784-95be-40af-98df-14e58bf411f0-crio-socket\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.791455 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5c0784-95be-40af-98df-14e58bf411f0-data-volume\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.791455 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d5c0784-95be-40af-98df-14e58bf411f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.791543 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b10257bd-92be-4617-beab-f2270c5d396d-metrics-tls\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.791573 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791543 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc79x\" (UniqueName: \"kubernetes.io/projected/889ada6d-d4ee-4f5e-aa22-23637608940a-kube-api-access-zc79x\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:05.791600 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10257bd-92be-4617-beab-f2270c5d396d-tmp-dir\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.791600 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxbjn\" (UniqueName: \"kubernetes.io/projected/7d5c0784-95be-40af-98df-14e58bf411f0-kube-api-access-vxbjn\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.791675 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b10257bd-92be-4617-beab-f2270c5d396d-config-volume\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.791705 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/889ada6d-d4ee-4f5e-aa22-23637608940a-cert\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:05.791735 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.791716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d5c0784-95be-40af-98df-14e58bf411f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.843370 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.843351 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 21:15:05.843475 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.843353 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 21:15:05.843475 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.843381 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 21:15:05.843475 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.843442 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 21:15:05.892589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/889ada6d-d4ee-4f5e-aa22-23637608940a-cert\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:05.892657 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d5c0784-95be-40af-98df-14e58bf411f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.892776 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7xd\" (UniqueName: \"kubernetes.io/projected/b10257bd-92be-4617-beab-f2270c5d396d-kube-api-access-bc7xd\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.892811 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d5c0784-95be-40af-98df-14e58bf411f0-crio-socket\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.892811 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5c0784-95be-40af-98df-14e58bf411f0-data-volume\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.892903 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d5c0784-95be-40af-98df-14e58bf411f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.892903 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b10257bd-92be-4617-beab-f2270c5d396d-metrics-tls\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.892903 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc79x\" (UniqueName: \"kubernetes.io/projected/889ada6d-d4ee-4f5e-aa22-23637608940a-kube-api-access-zc79x\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:05.892903 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10257bd-92be-4617-beab-f2270c5d396d-tmp-dir\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.893068 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxbjn\" (UniqueName: \"kubernetes.io/projected/7d5c0784-95be-40af-98df-14e58bf411f0-kube-api-access-vxbjn\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.893068 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.892959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b10257bd-92be-4617-beab-f2270c5d396d-config-volume\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.893068 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.893041 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d5c0784-95be-40af-98df-14e58bf411f0-crio-socket\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.893204 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.893117 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5c0784-95be-40af-98df-14e58bf411f0-data-volume\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.893259 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.893220 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d5c0784-95be-40af-98df-14e58bf411f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.893549 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.893529 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10257bd-92be-4617-beab-f2270c5d396d-tmp-dir\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.893793 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.893761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b10257bd-92be-4617-beab-f2270c5d396d-config-volume\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.896401 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.896376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/889ada6d-d4ee-4f5e-aa22-23637608940a-cert\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:05.896474 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.896399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d5c0784-95be-40af-98df-14e58bf411f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.896474 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.896399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b10257bd-92be-4617-beab-f2270c5d396d-metrics-tls\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.949484 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.949415 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7xd\" (UniqueName: \"kubernetes.io/projected/b10257bd-92be-4617-beab-f2270c5d396d-kube-api-access-bc7xd\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:05.949484 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.949433 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxbjn\" (UniqueName: \"kubernetes.io/projected/7d5c0784-95be-40af-98df-14e58bf411f0-kube-api-access-vxbjn\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:05.949904 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:05.949884 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc79x\" (UniqueName: \"kubernetes.io/projected/889ada6d-d4ee-4f5e-aa22-23637608940a-kube-api-access-zc79x\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:06.000450 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.000422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:06.005623 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.005595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:06.013293 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.013275 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:06.384713 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.384680 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9srkw"] Nov 03 21:15:06.388823 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:06.388790 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10257bd_92be_4617_beab_f2270c5d396d.slice/crio-ca9c819ac2c4e0f7a68114381873732c967e7b6f45c794dd45031758fcd8b44d WatchSource:0}: Error finding container ca9c819ac2c4e0f7a68114381873732c967e7b6f45c794dd45031758fcd8b44d: Status 404 returned error can't find the container with id ca9c819ac2c4e0f7a68114381873732c967e7b6f45c794dd45031758fcd8b44d Nov 03 21:15:06.435357 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.435328 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thpr2"] Nov 03 21:15:06.437866 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.437836 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xcf7l"] Nov 03 21:15:06.439083 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:06.439057 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889ada6d_d4ee_4f5e_aa22_23637608940a.slice/crio-2de3fa79e5c8b33275c84f5724425e8da7962e2d41a7f67406e766d06fc9c3d9 WatchSource:0}: Error finding container 2de3fa79e5c8b33275c84f5724425e8da7962e2d41a7f67406e766d06fc9c3d9: Status 404 returned error can't find the container with id 2de3fa79e5c8b33275c84f5724425e8da7962e2d41a7f67406e766d06fc9c3d9 Nov 03 21:15:06.441888 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:06.441869 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5c0784_95be_40af_98df_14e58bf411f0.slice/crio-5e7e0019da9070b86d795993d42d59571cf48906339613bc923976fb657b277c WatchSource:0}: Error finding container 5e7e0019da9070b86d795993d42d59571cf48906339613bc923976fb657b277c: Status 404 returned error can't find the container with id 5e7e0019da9070b86d795993d42d59571cf48906339613bc923976fb657b277c Nov 03 21:15:06.776815 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.776738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:06.777235 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.776834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:06.777235 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.776863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:06.829064 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.829034 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:15:06.829195 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.829179 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:15:06.829254 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.829195 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\"" Nov 03 21:15:06.829305 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.829252 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\"" Nov 03 21:15:06.829305 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.829262 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:15:06.829400 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.829334 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:15:06.829400 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.829359 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\"" Nov 03 21:15:06.996208 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.996176 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcf7l" event={"ID":"7d5c0784-95be-40af-98df-14e58bf411f0","Type":"ContainerStarted","Data":"cd597a31c127369ce9173d2056146ca9a095c4b09278dba07c063b4ec35b4053"} Nov 03 21:15:06.996370 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.996232 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcf7l" event={"ID":"7d5c0784-95be-40af-98df-14e58bf411f0","Type":"ContainerStarted","Data":"5e7e0019da9070b86d795993d42d59571cf48906339613bc923976fb657b277c"} Nov 03 21:15:06.997333 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.997313 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thpr2" event={"ID":"889ada6d-d4ee-4f5e-aa22-23637608940a","Type":"ContainerStarted","Data":"2de3fa79e5c8b33275c84f5724425e8da7962e2d41a7f67406e766d06fc9c3d9"} Nov 03 21:15:06.998260 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:06.998241 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9srkw" event={"ID":"b10257bd-92be-4617-beab-f2270c5d396d","Type":"ContainerStarted","Data":"ca9c819ac2c4e0f7a68114381873732c967e7b6f45c794dd45031758fcd8b44d"} Nov 03 21:15:07.033334 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:07.033274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:08.003452 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:08.003419 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcf7l" event={"ID":"7d5c0784-95be-40af-98df-14e58bf411f0","Type":"ContainerStarted","Data":"83f2dc95f20d54875350cd746337758d15010794c942b452774d8bfde8fa6bda"} Nov 03 21:15:08.314096 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:08.314024 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:08.550177 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:08.550126 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:08.803828 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:08.803792 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:09.008949 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.008871 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9srkw" event={"ID":"b10257bd-92be-4617-beab-f2270c5d396d","Type":"ContainerStarted","Data":"b9eaff2a71106d4d96c6fdb1ff1ddae07f468116ead0115ebc586cbe2fb51a64"} Nov 03 21:15:09.008949 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.008932 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9srkw" event={"ID":"b10257bd-92be-4617-beab-f2270c5d396d","Type":"ContainerStarted","Data":"3585a92f53bb4d1bed86e79af92535625b6ed440a89770068585552033643314"} Nov 03 21:15:09.009429 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.008967 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:09.122094 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.122035 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9srkw" podStartSLOduration=1.9239977339999998 podStartE2EDuration="4.122017626s" totalImagesPullingTime="2.198019892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:09.121791249 +0000 UTC m=+49.855055056" watchObservedRunningTime="2026-11-03 21:15:09.122017626 +0000 UTC m=+49.855281400" Nov 03 21:15:09.266102 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.266012 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-789d48d68-988j4"] Nov 03 21:15:09.267955 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.267931 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.318670 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.318635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.318842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.318698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.318842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.318745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.318842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.318767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9tj\" (UniqueName: \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.318842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.318804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.318842 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.318831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.319080 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.318878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.321342 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.321320 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 21:15:09.321488 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.321381 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 21:15:09.321488 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.321413 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kz2rr\"" Nov 03 21:15:09.321488 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.321445 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 21:15:09.321488 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.321462 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 21:15:09.321702 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.321502 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 21:15:09.324354 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.324331 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 21:15:09.324453 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.324390 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 21:15:09.325323 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.325307 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789d48d68-988j4"] Nov 03 21:15:09.419777 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.419735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.419951 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.419813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.419951 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.419852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.419951 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.419888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.419951 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.419935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.420117 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.419959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9tj\" (UniqueName: \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.420117 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.419994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.420875 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.420849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.421186 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.421159 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.421364 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.421345 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.421841 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.421813 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.422889 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.422863 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.424313 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.424296 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.477250 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.477218 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9tj\" (UniqueName: \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.581110 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.581087 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:09.950701 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:09.950669 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789d48d68-988j4"] Nov 03 21:15:09.995097 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:09.995068 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50541dcd_b16b_493b_9675_0f232b6abb4e.slice/crio-9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e WatchSource:0}: Error finding container 9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e: Status 404 returned error can't find the container with id 9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e Nov 03 21:15:10.013175 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:10.013147 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thpr2" event={"ID":"889ada6d-d4ee-4f5e-aa22-23637608940a","Type":"ContainerStarted","Data":"46b735bc7a3b16ed5f7070e9b54a388c0768cfb6ca5b04cf2cc8797b857c2cd9"} Nov 03 21:15:10.015446 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:10.015417 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcf7l" event={"ID":"7d5c0784-95be-40af-98df-14e58bf411f0","Type":"ContainerStarted","Data":"09116edb3f11399dcb30fdad986e0642301afcaf5f39299973e5b90ccd5b6562"} Nov 03 21:15:10.016730 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:10.016706 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789d48d68-988j4" event={"ID":"50541dcd-b16b-493b-9675-0f232b6abb4e","Type":"ContainerStarted","Data":"9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e"} Nov 03 21:15:10.269417 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:10.269331 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:11.080495 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:11.080453 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:11.126820 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:11.126762 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-thpr2" podStartSLOduration=3.036695003 podStartE2EDuration="6.126716679s" totalImagesPullingTime="3.090021676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:11.013220511 +0000 UTC m=+51.746484285" watchObservedRunningTime="2026-11-03 21:15:11.126716679 +0000 UTC m=+51.859980452" Nov 03 21:15:11.127680 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:11.127645 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xcf7l" podStartSLOduration=3.192070853 podStartE2EDuration="6.127633717s" totalImagesPullingTime="2.935562864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:15:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:11.126232336 +0000 UTC m=+51.859496109" watchObservedRunningTime="2026-11-03 21:15:11.127633717 +0000 UTC m=+51.860897483" Nov 03 21:15:11.302215 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:11.302180 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:13.027422 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:13.027384 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789d48d68-988j4" event={"ID":"50541dcd-b16b-493b-9675-0f232b6abb4e","Type":"ContainerStarted","Data":"be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e"} Nov 03 21:15:13.150871 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:13.150822 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-789d48d68-988j4" podStartSLOduration=1.2175375800000001 podStartE2EDuration="4.15080778s" totalImagesPullingTime="2.9332702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:13.150683709 +0000 UTC m=+53.883947480" watchObservedRunningTime="2026-11-03 21:15:13.15080778 +0000 UTC m=+53.884071566" Nov 03 21:15:13.608243 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:13.608211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:13.961720 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:13.961639 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:14.590727 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:14.590691 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:14.862007 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:14.861927 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:15.086337 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.086301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:15.159423 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.159351 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t"] Nov 03 21:15:15.161305 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.161292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.213692 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.213661 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 21:15:15.213838 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.213739 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 21:15:15.263226 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrsz\" (UniqueName: \"kubernetes.io/projected/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-kube-api-access-kvrsz\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.263226 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263219 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 21:15:15.263404 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.263404 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263240 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 21:15:15.263404 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.263404 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263238 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:15:15.263404 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263240 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-46jbj\"" Nov 03 21:15:15.263404 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.263339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.264192 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.264176 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t"] Nov 03 21:15:15.271390 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.271369 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb"] Nov 03 21:15:15.273342 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.273329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.314490 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.314469 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tbv9l"] Nov 03 21:15:15.316176 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.316163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.324377 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.324355 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:15.325240 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.325221 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 21:15:15.325240 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.325240 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-95lxt\"" Nov 03 21:15:15.325370 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.325267 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:15:15.325370 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.325325 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 21:15:15.332476 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.332456 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb"] Nov 03 21:15:15.363827 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.363803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-textfile\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.363937 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.363829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-metrics-client-ca\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.363937 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.363860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.363937 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.363888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-root\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.364042 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.363964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrsz\" (UniqueName: \"kubernetes.io/projected/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-kube-api-access-kvrsz\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.364042 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.363994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-sys\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.364042 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.364042 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/befe88bf-8d8b-47b8-a244-06524e4c3e28-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.364165 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.364165 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpr2d\" (UniqueName: \"kubernetes.io/projected/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-kube-api-access-vpr2d\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.364235 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fld\" (UniqueName: \"kubernetes.io/projected/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-api-access-p9fld\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.364270 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.364270 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.364332 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.364373 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.364434 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.364470 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-tls\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.364502 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-wtmp\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.364548 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.364800 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.364774 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.366817 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.366790 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.366897 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.366832 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.367934 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.367918 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 21:15:15.368009 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.367964 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 21:15:15.368053 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.368031 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79p29\"" Nov 03 21:15:15.423207 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.423150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrsz\" (UniqueName: \"kubernetes.io/projected/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-kube-api-access-kvrsz\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.425506 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.425488 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 21:15:15.465722 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.465832 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpr2d\" (UniqueName: \"kubernetes.io/projected/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-kube-api-access-vpr2d\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.465832 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fld\" (UniqueName: \"kubernetes.io/projected/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-api-access-p9fld\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.465832 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.465832 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.465999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.465999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-tls\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.465999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-wtmp\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.465999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.465999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-textfile\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.465999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.465966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-metrics-client-ca\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.466269 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.466269 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-root\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.466269 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-sys\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.466269 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/befe88bf-8d8b-47b8-a244-06524e4c3e28-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.466269 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466222 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-wtmp\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.466503 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466362 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-textfile\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.466503 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/befe88bf-8d8b-47b8-a244-06524e4c3e28-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.466792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466631 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.466792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-root\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.466792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.466702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-sys\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.467220 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.467193 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-metrics-client-ca\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.467348 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.467326 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.467579 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.467556 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.468459 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.468420 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.468562 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.468465 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.468562 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.468530 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-tls\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.468877 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.468857 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.472655 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.472641 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:15.522944 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.522885 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpr2d\" (UniqueName: \"kubernetes.io/projected/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-kube-api-access-vpr2d\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.523087 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.523068 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fld\" (UniqueName: \"kubernetes.io/projected/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-api-access-p9fld\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.555226 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.555199 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:15.584524 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.584502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:15.626999 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.626967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:15.639442 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:15.639418 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffd8c3e_675e_45b0_b22f_445e5a92bbfd.slice/crio-88ebbdb1e42452ff11f3007689a4e05014309f480269dc9a94aff80063132f55 WatchSource:0}: Error finding container 88ebbdb1e42452ff11f3007689a4e05014309f480269dc9a94aff80063132f55: Status 404 returned error can't find the container with id 88ebbdb1e42452ff11f3007689a4e05014309f480269dc9a94aff80063132f55 Nov 03 21:15:15.791482 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.791404 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:15.893706 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.893671 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t"] Nov 03 21:15:15.896980 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:15.896959 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7459ce_aa09_4c74_b5ef_8424cb8d24b5.slice/crio-d4b26dcce58bd3521ca66c870cdc6619c39ae14f7647ff9c1efffa33f9e476ac WatchSource:0}: Error finding container d4b26dcce58bd3521ca66c870cdc6619c39ae14f7647ff9c1efffa33f9e476ac: Status 404 returned error can't find the container with id d4b26dcce58bd3521ca66c870cdc6619c39ae14f7647ff9c1efffa33f9e476ac Nov 03 21:15:15.960159 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:15.960131 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb"] Nov 03 21:15:15.963218 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:15.963193 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefe88bf_8d8b_47b8_a244_06524e4c3e28.slice/crio-754f29b6870723d850df46b02b896181e6a4589d4c281a6def380fd04e7b7cd5 WatchSource:0}: Error finding container 754f29b6870723d850df46b02b896181e6a4589d4c281a6def380fd04e7b7cd5: Status 404 returned error can't find the container with id 754f29b6870723d850df46b02b896181e6a4589d4c281a6def380fd04e7b7cd5 Nov 03 21:15:16.016851 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.016824 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:16.039052 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.039017 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tbv9l" event={"ID":"fffd8c3e-675e-45b0-b22f-445e5a92bbfd","Type":"ContainerStarted","Data":"88ebbdb1e42452ff11f3007689a4e05014309f480269dc9a94aff80063132f55"} Nov 03 21:15:16.040152 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.040128 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" event={"ID":"befe88bf-8d8b-47b8-a244-06524e4c3e28","Type":"ContainerStarted","Data":"754f29b6870723d850df46b02b896181e6a4589d4c281a6def380fd04e7b7cd5"} Nov 03 21:15:16.041844 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.041797 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" event={"ID":"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5","Type":"ContainerStarted","Data":"2b7aa7e919419176494074b4692452c64dcede417247e4ee7c042a196183f5bc"} Nov 03 21:15:16.041844 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.041840 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" event={"ID":"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5","Type":"ContainerStarted","Data":"d029f8f190f6baabdb9047d4cedb71703a5d4861f33f82ec9de69d2502b7a2c0"} Nov 03 21:15:16.041942 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.041857 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" event={"ID":"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5","Type":"ContainerStarted","Data":"d4b26dcce58bd3521ca66c870cdc6619c39ae14f7647ff9c1efffa33f9e476ac"} Nov 03 21:15:16.314662 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.314570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:16.698424 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.698387 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:16.955222 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:16.955185 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:17.047765 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.047728 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tbv9l" event={"ID":"fffd8c3e-675e-45b0-b22f-445e5a92bbfd","Type":"ContainerStarted","Data":"97912fdef66d9d68f9492ddd36b596368caf8a0f04417617657ec921324a727c"} Nov 03 21:15:17.494425 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.494397 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:15:17.496908 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.496890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.549379 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549353 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 21:15:17.549379 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549372 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 21:15:17.549589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549391 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 21:15:17.549589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549380 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 21:15:17.549589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549395 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 21:15:17.549589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549414 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 21:15:17.549589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549434 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 21:15:17.549589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549447 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 21:15:17.549589 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.549514 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4ccvj\"" Nov 03 21:15:17.561664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.561641 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:15:17.561792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.561637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:17.584065 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584155 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584409 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584409 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584409 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584409 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584364 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584409 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584652 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584652 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584652 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584799 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584799 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.584799 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.584727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9kr\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.648446 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.648414 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 21:15:17.685447 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.685421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.685595 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.685457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.685595 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.685492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.685846 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.685813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.685964 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.685870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.685964 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.685900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9kr\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.685964 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.685955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.686121 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.686020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.686121 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.686057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.686121 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.686082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.686121 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.686116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.686296 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.686146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.686296 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.686176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.688722 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.687037 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.688722 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.688305 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.688722 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.688492 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.688722 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.688643 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.689489 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.689016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.689489 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.689035 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.689644 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.689522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.689927 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.689908 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.690742 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.690720 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.691124 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.691106 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.691226 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.691207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.692948 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.692928 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.743199 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.743174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9kr\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.812868 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.812839 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:17.832212 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.832173 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh"] Nov 03 21:15:17.834431 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.834404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.886934 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886758 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 21:15:17.886934 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886758 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 21:15:17.886934 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886861 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 21:15:17.886934 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886910 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qqtpl\"" Nov 03 21:15:17.886934 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887183 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886952 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 21:15:17.887183 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmvs\" (UniqueName: \"kubernetes.io/projected/58d5c977-b290-45f5-a219-8f2a98eab0f6-kube-api-access-kfmvs\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887183 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.886932 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 21:15:17.887183 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d5c977-b290-45f5-a219-8f2a98eab0f6-tmp\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887183 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d5c977-b290-45f5-a219-8f2a98eab0f6-metrics-client-ca\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887183 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887349 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887349 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887257 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-82cgq4d3kfu1j\"" Nov 03 21:15:17.887349 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887349 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-grpc-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.887456 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.887359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.895795 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.895776 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh"] Nov 03 21:15:17.988356 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988356 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988572 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988572 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-grpc-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988572 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988572 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988572 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmvs\" (UniqueName: \"kubernetes.io/projected/58d5c977-b290-45f5-a219-8f2a98eab0f6-kube-api-access-kfmvs\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988843 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d5c977-b290-45f5-a219-8f2a98eab0f6-tmp\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.988843 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.988652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d5c977-b290-45f5-a219-8f2a98eab0f6-metrics-client-ca\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.989171 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.989147 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d5c977-b290-45f5-a219-8f2a98eab0f6-tmp\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.989503 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.989462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d5c977-b290-45f5-a219-8f2a98eab0f6-metrics-client-ca\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.991604 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.991570 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.991702 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.991574 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.991702 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.991639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.991702 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.991676 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.991955 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.991938 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:17.991985 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:17.991955 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-grpc-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:18.044166 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.044139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmvs\" (UniqueName: \"kubernetes.io/projected/58d5c977-b290-45f5-a219-8f2a98eab0f6-kube-api-access-kfmvs\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:18.052884 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.052860 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" event={"ID":"befe88bf-8d8b-47b8-a244-06524e4c3e28","Type":"ContainerStarted","Data":"2ee8a8d57afde2111ac60f3d53b92e70687ae40694067f535f9d8d815a9dd516"} Nov 03 21:15:18.053000 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.052918 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" event={"ID":"befe88bf-8d8b-47b8-a244-06524e4c3e28","Type":"ContainerStarted","Data":"e9f9c715dc94fa3a60c401018eccc6f01047a0629c045472a56af9c040a44091"} Nov 03 21:15:18.053000 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.052945 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" event={"ID":"befe88bf-8d8b-47b8-a244-06524e4c3e28","Type":"ContainerStarted","Data":"571524152a5bc4f01c32214f9767b801d40f3d00ce3aabf87f3aedc0be5d2e44"} Nov 03 21:15:18.054782 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.054761 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" event={"ID":"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5","Type":"ContainerStarted","Data":"70a8c781b831ae41e5b90804ae05b2704a129f40fa0c15e76f60dc3e3ad469a8"} Nov 03 21:15:18.056017 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.055995 2575 generic.go:356] "Generic (PLEG): container finished" podID="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" containerID="97912fdef66d9d68f9492ddd36b596368caf8a0f04417617657ec921324a727c" exitCode=0 Nov 03 21:15:18.056087 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.056029 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tbv9l" event={"ID":"fffd8c3e-675e-45b0-b22f-445e5a92bbfd","Type":"ContainerDied","Data":"97912fdef66d9d68f9492ddd36b596368caf8a0f04417617657ec921324a727c"} Nov 03 21:15:18.150743 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.150721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:18.168048 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.168001 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" podStartSLOduration=1.551962984 podStartE2EDuration="3.167987738s" totalImagesPullingTime="1.616024754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:18.166119553 +0000 UTC m=+58.899383327" watchObservedRunningTime="2026-11-03 21:15:18.167987738 +0000 UTC m=+58.901251509" Nov 03 21:15:18.239177 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.239049 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:15:18.243994 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:18.243963 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9816356_82fa_42f8_b5b4_b5fd84a32042.slice/crio-47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c WatchSource:0}: Error finding container 47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c: Status 404 returned error can't find the container with id 47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c Nov 03 21:15:18.281008 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.280949 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" podStartSLOduration=1.88042314 podStartE2EDuration="3.280928854s" totalImagesPullingTime="1.400505714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:18.280319058 +0000 UTC m=+59.013582830" watchObservedRunningTime="2026-11-03 21:15:18.280928854 +0000 UTC m=+59.014192627" Nov 03 21:15:18.529792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:18.529710 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh"] Nov 03 21:15:18.533328 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:18.533307 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d5c977_b290_45f5_a219_8f2a98eab0f6.slice/crio-ac349d5b6c64518f91c5c6b4eaf6c3862d9c749288e6e783c9c6e67adb7dc44e WatchSource:0}: Error finding container ac349d5b6c64518f91c5c6b4eaf6c3862d9c749288e6e783c9c6e67adb7dc44e: Status 404 returned error can't find the container with id ac349d5b6c64518f91c5c6b4eaf6c3862d9c749288e6e783c9c6e67adb7dc44e Nov 03 21:15:19.019134 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.019106 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:19.062478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.062444 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tbv9l" event={"ID":"fffd8c3e-675e-45b0-b22f-445e5a92bbfd","Type":"ContainerStarted","Data":"894fad7e835b948a9334e9dac468437c9a536535580f0c5ae0e5b45dbcf6cbc7"} Nov 03 21:15:19.062646 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.062489 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tbv9l" event={"ID":"fffd8c3e-675e-45b0-b22f-445e5a92bbfd","Type":"ContainerStarted","Data":"dabe6dde609617641190b6b3b75933faf6176edaddee7a980fadedc883c5d482"} Nov 03 21:15:19.063664 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.063641 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" event={"ID":"58d5c977-b290-45f5-a219-8f2a98eab0f6","Type":"ContainerStarted","Data":"ac349d5b6c64518f91c5c6b4eaf6c3862d9c749288e6e783c9c6e67adb7dc44e"} Nov 03 21:15:19.064725 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.064702 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerStarted","Data":"47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c"} Nov 03 21:15:19.257932 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.257862 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tbv9l" podStartSLOduration=2.8636854940000003 podStartE2EDuration="4.25784252s" totalImagesPullingTime="1.305010788s" totalInitContainerRuntime="89.146238ms" isStatefulPod=true podCreationTimestamp="2026-11-03 21:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:19.247071724 +0000 UTC m=+59.980335499" watchObservedRunningTime="2026-11-03 21:15:19.25784252 +0000 UTC m=+59.991106294" Nov 03 21:15:19.407290 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.407261 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:19.581555 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.581502 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:19.581758 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.581633 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:19.587635 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.587589 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:19.633284 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.633247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:19.811427 ip-10-0-128-179 kubenswrapper[2575]: E1103 21:15:19.811385 2575 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9816356_82fa_42f8_b5b4_b5fd84a32042.slice/crio-conmon-9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9816356_82fa_42f8_b5b4_b5fd84a32042.slice/crio-9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:15:19.858663 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:19.858630 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:20.069971 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.069902 2575 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd" exitCode=0 Nov 03 21:15:20.070395 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.069999 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd"} Nov 03 21:15:20.074101 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.074084 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:20.083989 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.083949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:20.144387 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.144360 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-ff498dd48-lmxq5"] Nov 03 21:15:20.146787 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.146768 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.199118 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.199090 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 21:15:20.199251 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.199161 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 21:15:20.199251 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.199177 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 21:15:20.199251 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.199219 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 21:15:20.199364 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.199252 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4hvm9\"" Nov 03 21:15:20.199364 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.199289 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cue34h59ah6oo\"" Nov 03 21:15:20.214410 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.214380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-metrics-server-audit-profiles\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.214564 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.214538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-client-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.214682 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.214576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-client-certs\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.214682 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.214625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6rw\" (UniqueName: \"kubernetes.io/projected/f070dd5b-30ed-4531-8cf6-196779398aad-kube-api-access-tz6rw\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.214787 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.214712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.214787 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.214749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f070dd5b-30ed-4531-8cf6-196779398aad-audit-log\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.214880 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.214841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-tls\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.241493 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.241462 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-ff498dd48-lmxq5"] Nov 03 21:15:20.316373 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.316337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-client-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.316548 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.316386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-client-certs\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.316548 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.316417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6rw\" (UniqueName: \"kubernetes.io/projected/f070dd5b-30ed-4531-8cf6-196779398aad-kube-api-access-tz6rw\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.316548 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.316467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.316548 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.316501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f070dd5b-30ed-4531-8cf6-196779398aad-audit-log\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.316548 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.316543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-tls\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.316781 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.316574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-metrics-server-audit-profiles\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.317829 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.317791 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-metrics-server-audit-profiles\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.318595 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.318571 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f070dd5b-30ed-4531-8cf6-196779398aad-audit-log\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.318874 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.318774 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.321080 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.320992 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-client-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.321722 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.321700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-tls\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.323214 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.323188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-client-certs\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.375585 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.375560 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6rw\" (UniqueName: \"kubernetes.io/projected/f070dd5b-30ed-4531-8cf6-196779398aad-kube-api-access-tz6rw\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.459447 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.459413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:20.471993 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.471950 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:20.737760 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.737681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:20.888143 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.888074 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-ff498dd48-lmxq5"] Nov 03 21:15:20.896853 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:20.896821 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf070dd5b_30ed_4531_8cf6_196779398aad.slice/crio-793602ed89a001b62c8b5158c1055fb18c460d287078f21ee9de3d461e41fda3 WatchSource:0}: Error finding container 793602ed89a001b62c8b5158c1055fb18c460d287078f21ee9de3d461e41fda3: Status 404 returned error can't find the container with id 793602ed89a001b62c8b5158c1055fb18c460d287078f21ee9de3d461e41fda3 Nov 03 21:15:20.996715 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:20.996649 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:21.077131 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:21.076991 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" event={"ID":"58d5c977-b290-45f5-a219-8f2a98eab0f6","Type":"ContainerStarted","Data":"e73fefb96bab1e0e9c3a326cd70d53e154c4a5fe7f0161ba22a00275b7d48dc5"} Nov 03 21:15:21.077131 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:21.077088 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" event={"ID":"58d5c977-b290-45f5-a219-8f2a98eab0f6","Type":"ContainerStarted","Data":"f2d52d95d18e7704483fd54afa4dffcd2434c4f9ff2ccc9082105907f0371a54"} Nov 03 21:15:21.078824 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:21.078796 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" event={"ID":"f070dd5b-30ed-4531-8cf6-196779398aad","Type":"ContainerStarted","Data":"793602ed89a001b62c8b5158c1055fb18c460d287078f21ee9de3d461e41fda3"} Nov 03 21:15:22.086504 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:22.086469 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerStarted","Data":"e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a"} Nov 03 21:15:22.086962 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:22.086556 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerStarted","Data":"11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d"} Nov 03 21:15:22.090596 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:22.090573 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" event={"ID":"58d5c977-b290-45f5-a219-8f2a98eab0f6","Type":"ContainerStarted","Data":"9ef34f28bee52295304ae1e1e9cd788e73f1a3e94dc8836080404b1d09080810"} Nov 03 21:15:23.095703 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.095662 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" event={"ID":"f070dd5b-30ed-4531-8cf6-196779398aad","Type":"ContainerStarted","Data":"4490b6b654dc04e35cff6c95d2f35392825e2216ad26d835b37f2d86e4ef25b6"} Nov 03 21:15:23.098716 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.098576 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerStarted","Data":"2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb"} Nov 03 21:15:23.098716 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.098667 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerStarted","Data":"6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00"} Nov 03 21:15:23.098716 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.098706 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerStarted","Data":"1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c"} Nov 03 21:15:23.101928 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.101799 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" event={"ID":"58d5c977-b290-45f5-a219-8f2a98eab0f6","Type":"ContainerStarted","Data":"d35873234e0d823dc8847730b0d865a1ad3000547f412ad3cf8ef85d29f37ffa"} Nov 03 21:15:23.101928 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.101847 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" event={"ID":"58d5c977-b290-45f5-a219-8f2a98eab0f6","Type":"ContainerStarted","Data":"0bbf8c19c40494df270d0a52cf93b3427580e0da62bec4ccaf99662c4ffbc16e"} Nov 03 21:15:23.101928 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.101871 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" event={"ID":"58d5c977-b290-45f5-a219-8f2a98eab0f6","Type":"ContainerStarted","Data":"f39f3a0aa860a88096fd80d1c92d7884e6d25c280f734c7b19c0824f60c3ccd8"} Nov 03 21:15:23.102074 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.101968 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:23.209865 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.209774 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" podStartSLOduration=1.175601301 podStartE2EDuration="3.209761511s" totalImagesPullingTime="2.03416021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:23.208300916 +0000 UTC m=+63.941564688" watchObservedRunningTime="2026-11-03 21:15:23.209761511 +0000 UTC m=+63.943025345" Nov 03 21:15:23.328821 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:23.328772 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" podStartSLOduration=2.727020269 podStartE2EDuration="6.328757279s" totalImagesPullingTime="3.60173701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:23.327167425 +0000 UTC m=+64.060431198" watchObservedRunningTime="2026-11-03 21:15:23.328757279 +0000 UTC m=+64.062021050" Nov 03 21:15:24.109027 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.108989 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerStarted","Data":"aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42"} Nov 03 21:15:24.313514 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.313444 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.213020049 podStartE2EDuration="7.313425118s" totalImagesPullingTime="4.084820895s" totalInitContainerRuntime="15.584174ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:24.282785213 +0000 UTC m=+65.016048984" watchObservedRunningTime="2026-11-03 21:15:24.313425118 +0000 UTC m=+65.046688890" Nov 03 21:15:24.364359 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.364292 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:24.368334 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.368308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.421208 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421179 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 21:15:24.421353 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421236 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 21:15:24.421353 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421307 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 21:15:24.421467 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421445 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 21:15:24.421467 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421452 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 21:15:24.421765 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421750 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 21:15:24.421853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421783 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 21:15:24.421853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421829 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 21:15:24.421853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421837 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 21:15:24.421853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421833 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 21:15:24.421853 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421847 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 21:15:24.421995 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.421874 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ftm6x\"" Nov 03 21:15:24.442982 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.442947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:24.459146 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459260 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459260 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459260 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459361 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459361 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459361 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459450 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459450 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459511 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459543 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459583 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvvz\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459640 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459683 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459683 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459745 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459778 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459810 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459844 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.459844 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.459833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.470465 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.470445 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 21:15:24.470651 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.470445 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-14esuk0guf2o7\"" Nov 03 21:15:24.520358 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.520288 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 21:15:24.520501 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.520390 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 21:15:24.522003 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.521984 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:24.560706 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.560683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.560784 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.560713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvvz\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.560784 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.560741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.560886 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.560783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.560948 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.560929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.560993 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.560960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561047 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561047 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561152 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561152 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561152 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561152 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561331 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561331 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561331 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561331 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561331 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561585 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561585 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561585 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561389 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561585 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561781 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561741 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.561844 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.561782 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.562275 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.562247 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.563977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.564104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.564409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.564509 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.564521 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.564707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.564930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565193 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.565125 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565503 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.565216 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.565503 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.565400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.566658 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.566634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.566880 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.566860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.567172 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.567152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.567239 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.567221 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.567239 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.567226 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.617386 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.617329 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvvz\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:24.692076 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:24.692040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:25.174354 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.174320 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:25.178769 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:25.178747 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed2f3a9_343c_4a13_a9d8_68dafc4472fb.slice/crio-adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73 WatchSource:0}: Error finding container adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73: Status 404 returned error can't find the container with id adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73 Nov 03 21:15:25.574113 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.574045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:25.574113 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.574104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:25.625080 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.625053 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:15:25.625229 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.625131 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:15:25.636604 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.636583 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:25.636739 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.636645 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:25.674954 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.674925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:25.725724 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.725698 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:15:25.734972 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.734945 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:25.737330 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.737316 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\"" Nov 03 21:15:25.746517 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.746498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:25.748822 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.748806 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\"" Nov 03 21:15:25.758490 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.758474 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:25.786478 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.786450 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:15:25.801643 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.801440 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:25.972859 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:25.972827 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:26.042679 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.042652 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\"" Nov 03 21:15:26.052687 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.052667 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:26.121246 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.117473 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9vfbk"] Nov 03 21:15:26.123885 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.123850 2575 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="1e0348bb86e8b11e81b7d2aab96b40e40ad38da22be1d0969cd611165dee501e" exitCode=0 Nov 03 21:15:26.125791 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.124073 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"1e0348bb86e8b11e81b7d2aab96b40e40ad38da22be1d0969cd611165dee501e"} Nov 03 21:15:26.125860 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.125815 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerStarted","Data":"adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73"} Nov 03 21:15:26.128990 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:26.128968 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8acde6b_bb34_4c16_9afe_aa4c21f59dd6.slice/crio-868be30f7f75bfd2a455980daefa80e1d9f7a3232223d43f4c1a9e5a69003a7f WatchSource:0}: Error finding container 868be30f7f75bfd2a455980daefa80e1d9f7a3232223d43f4c1a9e5a69003a7f: Status 404 returned error can't find the container with id 868be30f7f75bfd2a455980daefa80e1d9f7a3232223d43f4c1a9e5a69003a7f Nov 03 21:15:26.166816 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.166793 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkqjv"] Nov 03 21:15:26.169581 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:26.169558 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49fe757_54a2_4871_b450_d48e15571c9c.slice/crio-1e68f3a539d2936c561295045e6f0c76e536ccf481c2fe8abfead99c1aead68f WatchSource:0}: Error finding container 1e68f3a539d2936c561295045e6f0c76e536ccf481c2fe8abfead99c1aead68f: Status 404 returned error can't find the container with id 1e68f3a539d2936c561295045e6f0c76e536ccf481c2fe8abfead99c1aead68f Nov 03 21:15:26.201830 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.201803 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:26.422137 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:26.422111 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2xkcf"] Nov 03 21:15:26.425142 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:26.425118 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f4aeef_6f5d_48bc_8bbe_bb7e8e5658de.slice/crio-7f9f5ec11211c230915c9646b71f79ebdac153a2ce637e63381703933c494c27 WatchSource:0}: Error finding container 7f9f5ec11211c230915c9646b71f79ebdac153a2ce637e63381703933c494c27: Status 404 returned error can't find the container with id 7f9f5ec11211c230915c9646b71f79ebdac153a2ce637e63381703933c494c27 Nov 03 21:15:27.134207 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:27.134129 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2xkcf" event={"ID":"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de","Type":"ContainerStarted","Data":"7f9f5ec11211c230915c9646b71f79ebdac153a2ce637e63381703933c494c27"} Nov 03 21:15:27.135763 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:27.135736 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkqjv" event={"ID":"a49fe757-54a2-4871-b450-d48e15571c9c","Type":"ContainerStarted","Data":"1e68f3a539d2936c561295045e6f0c76e536ccf481c2fe8abfead99c1aead68f"} Nov 03 21:15:27.137540 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:27.137500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9vfbk" event={"ID":"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6","Type":"ContainerStarted","Data":"868be30f7f75bfd2a455980daefa80e1d9f7a3232223d43f4c1a9e5a69003a7f"} Nov 03 21:15:27.330716 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:27.330581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:27.563042 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:27.562893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:27.798541 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:27.798497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:28.145132 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.145092 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkqjv" event={"ID":"a49fe757-54a2-4871-b450-d48e15571c9c","Type":"ContainerStarted","Data":"f819626a7a0c0867de33b6f53935be1e348ce1bd56a30e4ffcefc21eec1f5b1f"} Nov 03 21:15:28.365523 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.365488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:28.476886 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.476783 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9797677c8-8gnvt"] Nov 03 21:15:28.481372 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.480782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.585410 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.585376 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9797677c8-8gnvt"] Nov 03 21:15:28.585792 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.585752 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 21:15:28.599119 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.599086 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:28.604422 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.604536 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.604536 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.604536 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.604536 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.604768 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmw5\" (UniqueName: \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.604768 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.604768 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.604633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.705525 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.705525 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.705778 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.705778 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.705778 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.705778 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmw5\" (UniqueName: \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.705778 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.706055 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.705789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.706055 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.706035 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.706351 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.706326 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.706744 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.706542 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.706965 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.706942 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.707300 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.707275 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.713433 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.713390 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.717275 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.717250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.764084 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.764018 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmw5\" (UniqueName: \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:28.795978 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:28.795949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:29.117587 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:29.117559 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:29.151170 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:29.151131 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkqjv" event={"ID":"a49fe757-54a2-4871-b450-d48e15571c9c","Type":"ContainerStarted","Data":"164d2de51b1ae617538922bda902472886168ae4445dd4b0ae49332b85c6cd13"} Nov 03 21:15:29.195154 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:29.195075 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9797677c8-8gnvt"] Nov 03 21:15:29.209338 ip-10-0-128-179 kubenswrapper[2575]: W1103 21:15:29.209306 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec556062_fda8_4a75_aac7_fb519d35285d.slice/crio-97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef WatchSource:0}: Error finding container 97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef: Status 404 returned error can't find the container with id 97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef Nov 03 21:15:29.360250 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:29.359046 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nkqjv" podStartSLOduration=67.564730825 podStartE2EDuration="1m9.359028322s" totalImagesPullingTime="1.794297497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:29.357409885 +0000 UTC m=+70.090673659" watchObservedRunningTime="2026-11-03 21:15:29.359028322 +0000 UTC m=+70.092292097" Nov 03 21:15:30.160060 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:30.160019 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9797677c8-8gnvt" event={"ID":"ec556062-fda8-4a75-aac7-fb519d35285d","Type":"ContainerStarted","Data":"394dcc9d384692b839ed50befe02f059de2f8fdfbe6689a3bf137a152e1c0501"} Nov 03 21:15:30.160539 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:30.160069 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9797677c8-8gnvt" event={"ID":"ec556062-fda8-4a75-aac7-fb519d35285d","Type":"ContainerStarted","Data":"97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef"} Nov 03 21:15:30.276460 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:30.276382 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9797677c8-8gnvt" podStartSLOduration=2.276365576 podStartE2EDuration="2.276365576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:30.276198394 +0000 UTC m=+71.009462167" watchObservedRunningTime="2026-11-03 21:15:30.276365576 +0000 UTC m=+71.009629345" Nov 03 21:15:30.764921 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:30.764890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:31.003845 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.003799 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:31.175840 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.175807 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerStarted","Data":"c8ac7fad04959805aef82621347439cfe06af1b68d2925a8187835b7ce95c333"} Nov 03 21:15:31.176376 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.176355 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerStarted","Data":"18a30cd7dc300c546d6acad101776849033be8d61c2af4e549b402de8e2626b9"} Nov 03 21:15:31.176506 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.176477 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerStarted","Data":"4f9dc7f4c4b16bf502a27e1482295444e6ca2e6b1cea68b03873d385a7ea3c52"} Nov 03 21:15:31.176653 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.176523 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerStarted","Data":"85bdc4954edc07d2accef1f8e7d806c92f99bad57a713dea15fd3b973511da87"} Nov 03 21:15:31.176653 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.176560 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerStarted","Data":"48a0135fd7b35e2f309129d181404c5b11b07beea0316520a87de39467bc245c"} Nov 03 21:15:31.176653 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.176605 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerStarted","Data":"b65817f7be82b8a646f4808732af1ae04b9e6a1921fced7d52f1a6168a41721a"} Nov 03 21:15:31.303936 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.303889 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.817177197 podStartE2EDuration="7.303873464s" totalImagesPullingTime="4.468760943s" totalInitContainerRuntime="17.935324ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:31.300370502 +0000 UTC m=+72.033634273" watchObservedRunningTime="2026-11-03 21:15:31.303873464 +0000 UTC m=+72.037137233" Nov 03 21:15:31.415326 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:31.415228 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2xkcf" podStartSLOduration=66.749798797 podStartE2EDuration="1m11.415212975s" totalImagesPullingTime="4.665414178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:31.414485912 +0000 UTC m=+72.147749682" watchObservedRunningTime="2026-11-03 21:15:31.415212975 +0000 UTC m=+72.148476746" Nov 03 21:15:32.183688 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:32.183650 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9vfbk" event={"ID":"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6","Type":"ContainerStarted","Data":"8cadd309995077ebbafbb1572f3d1a033decea6db1aa8104635c4129fa5141a2"} Nov 03 21:15:32.186683 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:32.186653 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2xkcf" event={"ID":"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de","Type":"ContainerStarted","Data":"3c9d5aa20aa626ed9464fd7d0eb63f4504a09f55c270752960a0b13b97e2c3fd"} Nov 03 21:15:32.187406 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:32.187391 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:32.296084 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:32.296021 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9vfbk" podStartSLOduration=66.362397373 podStartE2EDuration="1m12.296001638s" totalImagesPullingTime="5.933604265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:32.295382824 +0000 UTC m=+73.028646597" watchObservedRunningTime="2026-11-03 21:15:32.296001638 +0000 UTC m=+73.029265410" Nov 03 21:15:32.321293 ip-10-0-128-179 kubenswrapper[2575]: I1103 21:15:32.321174 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:15:32.321325 ip-10-0-128-179 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 21:15:32.336989 ip-10-0-128-179 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 21:15:32.337191 ip-10-0-128-179 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 21:15:32.337253 ip-10-0-128-179 systemd[1]: kubelet.service: Consumed 4.580s CPU time. Nov 03 21:15:32.356405 ip-10-0-128-179 systemd[1]: Starting Kubernetes Kubelet... Nov 03 21:15:32.417557 ip-10-0-128-179 kubenswrapper[5931]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:15:32.417557 ip-10-0-128-179 kubenswrapper[5931]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 21:15:32.417557 ip-10-0-128-179 kubenswrapper[5931]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:15:32.417557 ip-10-0-128-179 kubenswrapper[5931]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:15:32.419577 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419495 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.419577 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419578 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419582 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419585 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419588 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419591 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419594 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419596 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419599 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419602 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419604 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419623 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419626 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419628 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419631 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419634 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419637 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419641 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419646 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419655 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.419657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419658 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419662 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419665 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419667 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419671 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419673 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419676 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419679 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419681 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419684 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419686 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419689 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419691 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419694 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419697 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419700 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419702 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419705 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419708 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419711 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.420090 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419714 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419716 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419719 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419721 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419724 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419727 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419730 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419733 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419737 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419740 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419743 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419746 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419748 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419751 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419754 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419757 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419760 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419763 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419765 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.420542 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419769 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419771 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419774 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419777 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419780 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419783 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419785 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419788 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419791 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419793 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419796 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419799 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419802 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419804 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419807 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419809 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419812 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419814 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419816 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.421010 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419819 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419821 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419824 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419826 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419829 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419831 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419834 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419836 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419839 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419842 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419845 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419847 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419850 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419853 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419855 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419858 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419861 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419863 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419866 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.421452 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419869 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419871 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419874 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419876 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419879 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419882 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419887 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419890 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.419893 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420629 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420636 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420639 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420642 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420644 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420647 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420650 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420653 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420655 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420658 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420660 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.421922 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420663 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420666 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420669 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420672 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420674 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420678 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420680 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420683 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420685 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420688 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420691 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420693 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420696 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420698 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420701 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420703 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420706 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420708 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420711 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.422381 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420720 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420723 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420725 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420728 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420730 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420733 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420735 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420738 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420741 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420743 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420746 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420748 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420751 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420753 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420758 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420761 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420764 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420767 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420769 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420772 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.422834 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420775 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420778 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420780 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420783 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420785 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420788 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420791 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420793 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420796 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420798 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420801 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420803 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420806 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420814 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420817 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420820 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420822 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420825 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420827 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420830 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.423330 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420832 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420835 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420838 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420841 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420843 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420845 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420849 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420854 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420858 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420861 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420864 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420867 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420869 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420872 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420874 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420877 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420879 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420882 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420884 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420887 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.423816 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420890 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420893 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420896 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420898 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420901 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420906 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420912 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420915 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420917 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420920 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420923 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420925 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420928 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420930 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420933 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.420935 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421446 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421451 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421459 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.424305 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421462 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421465 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421467 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421470 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421473 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421476 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421478 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421481 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421484 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421486 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421489 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421492 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421495 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421498 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421500 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421503 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421506 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421508 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421511 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421521 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.424758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421524 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421526 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421529 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421531 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421534 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421536 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421539 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421541 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421544 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421546 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421549 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421552 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421554 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421557 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421559 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421562 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421564 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421567 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421569 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421572 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.425216 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421578 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421580 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421583 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421585 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421588 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421590 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421593 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421596 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421598 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421601 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421605 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421625 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421636 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421639 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421641 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421644 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421647 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421649 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421652 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.425708 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421654 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421657 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421660 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421663 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421666 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421668 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421671 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421674 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421676 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421679 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421681 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421684 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421687 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421691 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421693 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421698 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421703 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421706 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421709 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.426158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421712 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421715 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421717 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421720 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421722 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421725 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421728 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421735 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421738 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421741 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421744 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421746 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421749 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421752 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421755 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421758 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421760 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421763 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421765 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421768 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.426860 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421770 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421773 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421775 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421777 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.421780 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421879 5931 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421887 5931 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421897 5931 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421901 5931 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421905 5931 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421909 5931 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421913 5931 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421917 5931 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421920 5931 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421923 5931 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421927 5931 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421930 5931 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421934 5931 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421937 5931 flags.go:64] FLAG: --cgroup-root="" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421939 5931 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421948 5931 flags.go:64] FLAG: --client-ca-file="" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421951 5931 flags.go:64] FLAG: --cloud-provider="external" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421954 5931 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421960 5931 flags.go:64] FLAG: --cluster-domain="" Nov 03 21:15:32.427432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421963 5931 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421966 5931 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421969 5931 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421972 5931 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421976 5931 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421979 5931 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421982 5931 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421986 5931 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421989 5931 flags.go:64] FLAG: --contention-profiling="false" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421992 5931 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421995 5931 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.421998 5931 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422001 5931 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422005 5931 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422010 5931 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422013 5931 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422016 5931 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422019 5931 flags.go:64] FLAG: --enable-server="true" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422021 5931 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422029 5931 flags.go:64] FLAG: --event-burst="100" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422032 5931 flags.go:64] FLAG: --event-qps="50" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422036 5931 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422039 5931 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422042 5931 flags.go:64] FLAG: --eviction-hard="" Nov 03 21:15:32.428009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422046 5931 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422049 5931 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422052 5931 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422055 5931 flags.go:64] FLAG: --eviction-soft="" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422058 5931 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422066 5931 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422069 5931 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422072 5931 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422075 5931 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422078 5931 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422081 5931 flags.go:64] FLAG: --feature-gates="" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422085 5931 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422088 5931 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422091 5931 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422094 5931 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422097 5931 flags.go:64] FLAG: --healthz-port="10248" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422100 5931 flags.go:64] FLAG: --help="false" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422103 5931 flags.go:64] FLAG: --hostname-override="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422107 5931 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422110 5931 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422113 5931 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422117 5931 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422122 5931 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 21:15:32.428604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422125 5931 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422128 5931 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422130 5931 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422133 5931 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422136 5931 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422139 5931 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422144 5931 flags.go:64] FLAG: --kube-reserved="" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422147 5931 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422150 5931 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422153 5931 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422155 5931 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422158 5931 flags.go:64] FLAG: --lock-file="" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422161 5931 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422164 5931 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422167 5931 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422179 5931 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422182 5931 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422185 5931 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422188 5931 flags.go:64] FLAG: --logging-format="text" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422191 5931 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422194 5931 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422197 5931 flags.go:64] FLAG: --manifest-url="" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422200 5931 flags.go:64] FLAG: --manifest-url-header="" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422204 5931 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422207 5931 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 21:15:32.429151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422211 5931 flags.go:64] FLAG: --max-pods="110" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422214 5931 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422217 5931 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422219 5931 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422222 5931 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422225 5931 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422229 5931 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422232 5931 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422239 5931 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422242 5931 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422245 5931 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422248 5931 flags.go:64] FLAG: --pod-cidr="" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422251 5931 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422255 5931 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422258 5931 flags.go:64] FLAG: --pods-per-core="0" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422261 5931 flags.go:64] FLAG: --port="10250" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422264 5931 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422267 5931 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-08b57ca8d6ac47f41" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422270 5931 flags.go:64] FLAG: --qos-reserved="" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422285 5931 flags.go:64] FLAG: --read-only-port="10255" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422288 5931 flags.go:64] FLAG: --register-node="true" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422291 5931 flags.go:64] FLAG: --register-with-taints="" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422298 5931 flags.go:64] FLAG: --registry-burst="10" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422307 5931 flags.go:64] FLAG: --registry-qps="5" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422310 5931 flags.go:64] FLAG: --reserved-cpus="" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422313 5931 flags.go:64] FLAG: --reserved-memory="" Nov 03 21:15:32.429726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422317 5931 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422320 5931 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422323 5931 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422325 5931 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422328 5931 flags.go:64] FLAG: --runonce="false" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422331 5931 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422334 5931 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422337 5931 flags.go:64] FLAG: --seccomp-default="false" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422340 5931 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422343 5931 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422346 5931 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422349 5931 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422353 5931 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422356 5931 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422359 5931 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422362 5931 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422365 5931 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422368 5931 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422371 5931 flags.go:64] FLAG: --system-cgroups="" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422375 5931 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422381 5931 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422384 5931 flags.go:64] FLAG: --tls-cert-file="" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422387 5931 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422394 5931 flags.go:64] FLAG: --tls-min-version="" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422397 5931 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 21:15:32.430315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422400 5931 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422402 5931 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422405 5931 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422408 5931 flags.go:64] FLAG: --v="2" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422412 5931 flags.go:64] FLAG: --version="false" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422422 5931 flags.go:64] FLAG: --vmodule="" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422426 5931 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422429 5931 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422588 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422591 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422595 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422597 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422600 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422603 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422605 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422621 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422624 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422627 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422629 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422634 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422637 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.430894 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422639 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422642 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422645 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422648 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422650 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422654 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422656 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422659 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422661 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422664 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422667 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422669 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422672 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422674 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422677 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422680 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422682 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422690 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422693 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422696 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.431365 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422698 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422701 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422703 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422706 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422708 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422711 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422714 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422716 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422718 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422721 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422725 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422727 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422730 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422732 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422735 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422738 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422740 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422744 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422747 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422749 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.431850 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422752 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422754 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422757 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422759 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422762 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422764 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422767 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422770 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422773 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422775 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422790 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422793 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422796 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422798 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422801 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422804 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422808 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422812 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422815 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.432302 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422818 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422821 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422824 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422828 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422831 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422834 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422837 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422840 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422843 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422845 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422849 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422852 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422855 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422857 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422860 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422862 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422865 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422868 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422870 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.432801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422873 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422875 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422878 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422881 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422883 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422891 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422895 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422898 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422902 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422905 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422909 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422912 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422914 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422917 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422919 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.433326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.422944 5931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422992 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.422998 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423000 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423003 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423006 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423008 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423011 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423014 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423017 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423019 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423022 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423025 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423027 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423030 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423033 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423035 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423038 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423040 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.433910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423043 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423045 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423048 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423050 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423058 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423062 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423064 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423067 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423069 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423072 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423075 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423079 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423082 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423085 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423088 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423092 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423095 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423097 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423100 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.434370 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423102 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423105 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423109 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423112 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423115 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423117 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423120 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423122 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423125 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423127 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423130 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423133 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423136 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423138 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423141 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423144 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423147 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423149 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423157 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423160 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.434822 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423163 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423166 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423168 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423171 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423173 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423176 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423178 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423181 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423185 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423187 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423190 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423192 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423195 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423197 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423201 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423204 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423206 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423209 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423211 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423214 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.435274 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423216 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423219 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423222 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423224 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423227 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423229 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423232 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423234 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423237 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423239 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423242 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423251 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423254 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423256 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423259 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423261 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423263 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423266 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423268 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423271 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.435770 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423276 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423278 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423281 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423283 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423286 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423289 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423293 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423296 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.423299 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.428126 5931 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.428143 5931 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.428162 5931 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.428167 5931 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428223 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428227 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428231 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428234 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428237 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428239 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428243 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.436258 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428246 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428249 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428251 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428254 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428270 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428275 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428278 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428282 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428284 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428287 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428290 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428293 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428295 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428298 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428301 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428303 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428306 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428309 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428311 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428314 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.436782 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428316 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428319 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428322 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428326 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428330 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428333 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428336 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428339 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428341 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428344 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428346 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428350 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428354 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428357 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428360 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428362 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428365 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428367 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428370 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.437323 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428373 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428376 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428379 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428382 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428385 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428388 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428390 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428393 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428395 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428398 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428401 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428403 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428406 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428408 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428411 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428413 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428416 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428418 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428421 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428423 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.437818 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428426 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428428 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428431 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428433 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428435 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428438 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428440 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428443 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428446 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428448 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428451 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428453 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428456 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428459 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428462 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428465 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428468 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428470 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428473 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428475 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.438287 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428478 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428480 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428483 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428485 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428488 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428490 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428493 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428495 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428498 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428501 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428503 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428506 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428508 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428511 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428514 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428516 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428519 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428521 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428523 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.438797 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428526 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.428552 5931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428738 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428744 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428747 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428751 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428754 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428758 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428761 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428764 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428766 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428769 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428771 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428774 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428776 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428779 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.439233 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428781 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428784 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428786 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428789 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428792 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428794 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428797 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428799 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428802 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428804 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428807 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428809 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428813 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428817 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428820 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428822 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428825 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428827 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428830 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428832 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.439598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428835 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428837 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428840 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428842 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428845 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428848 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428851 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428853 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428856 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428858 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428861 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428863 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428866 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428868 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428871 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428873 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428876 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428878 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428881 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.440318 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428883 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428886 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428888 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428891 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428893 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428896 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428898 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428901 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428903 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428906 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428908 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428910 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428913 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428915 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428918 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428921 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428923 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428926 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428930 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428932 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.440835 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428935 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428937 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428939 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428942 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428944 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428947 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428949 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428952 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428954 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428957 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428959 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428962 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428964 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428967 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428969 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428972 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428974 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428977 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428979 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.441647 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428982 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428985 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428987 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428990 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428992 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428995 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.428997 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429000 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429002 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429005 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429009 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429012 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429015 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429018 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.429040 5931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429072 5931 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:15:32.442313 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429076 5931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429079 5931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429082 5931 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429085 5931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429089 5931 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429092 5931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429095 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429097 5931 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429100 5931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429102 5931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429105 5931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429107 5931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429110 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429112 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429115 5931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429117 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429120 5931 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429123 5931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429125 5931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:15:32.442990 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429128 5931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429130 5931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429132 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429135 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429138 5931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429140 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429143 5931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429145 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429148 5931 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429150 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429153 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429156 5931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429158 5931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429161 5931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429164 5931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429167 5931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429169 5931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429172 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429174 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:15:32.443862 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429177 5931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429182 5931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429185 5931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429187 5931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429190 5931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429192 5931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429195 5931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429197 5931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429200 5931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429202 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429204 5931 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429207 5931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429209 5931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429212 5931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429214 5931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429217 5931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429219 5931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429222 5931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429224 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429227 5931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:15:32.444719 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429229 5931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429232 5931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429234 5931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429237 5931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429240 5931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429243 5931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429245 5931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429248 5931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429251 5931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429253 5931 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429255 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429258 5931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429260 5931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429265 5931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429269 5931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429271 5931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429274 5931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429276 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429279 5931 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429281 5931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:15:32.445631 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429284 5931 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429288 5931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429291 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429294 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429297 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429300 5931 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429302 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429305 5931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429308 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429311 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429313 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429316 5931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429318 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429321 5931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429323 5931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429326 5931 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429328 5931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429335 5931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429338 5931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:15:32.446490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429341 5931 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429343 5931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429346 5931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429349 5931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429352 5931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429354 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429362 5931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:32.429365 5931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.429422 5931 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.430373 5931 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.430426 5931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.430597 5931 server.go:1014] "Starting client certificate rotation" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.430753 5931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.430810 5931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:09:19 +0000 UTC" deadline="2028-08-01 05:04:40.512651958 +0000 UTC" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.430845 5931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15271h49m8.081810557s" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.431113 5931 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:15:32.447398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.432116 5931 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:15:32.448135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.433368 5931 log.go:25] "Validated CRI v1 runtime API" Nov 03 21:15:32.448135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.435311 5931 log.go:25] "Validated CRI v1 image API" Nov 03 21:15:32.448135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.436027 5931 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 21:15:32.448135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.438281 5931 fs.go:135] Filesystem UUIDs: map[37c83c4d-ce9f-4d61-a220-ca63fe66b4c1:/dev/nvme0n1p3 799d1d18-ac6b-4e44-9d3d-5913512821c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 21:15:32.448359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.438299 5931 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/153b1abd7c935b1a467680a61676c9e0d4c288e22d517c904a18cea07a2edacf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/153b1abd7c935b1a467680a61676c9e0d4c288e22d517c904a18cea07a2edacf/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e68f3a539d2936c561295045e6f0c76e536ccf481c2fe8abfead99c1aead68f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e68f3a539d2936c561295045e6f0c76e536ccf481c2fe8abfead99c1aead68f/userdata/shm major:0 minor:447 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f0f1d254373a18301419cbd223a7bc66a5878b43e4cd615e1bee8a51f8644be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f0f1d254373a18301419cbd223a7bc66a5878b43e4cd615e1bee8a51f8644be/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2de3fa79e5c8b33275c84f5724425e8da7962e2d41a7f67406e766d06fc9c3d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2de3fa79e5c8b33275c84f5724425e8da7962e2d41a7f67406e766d06fc9c3d9/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38c585b403cc8fb42f8498fa7259ece6fb9a1137be5069db08c526f58f27ade8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38c585b403cc8fb42f8498fa7259ece6fb9a1137be5069db08c526f58f27ade8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51c80b41da04d7ba26386c909d3b07e93a25cc458b0d82ffc9cde5d8efa04452/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51c80b41da04d7ba26386c909d3b07e93a25cc458b0d82ffc9cde5d8efa04452/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e7e0019da9070b86d795993d42d59571cf48906339613bc923976fb657b277c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e7e0019da9070b86d795993d42d59571cf48906339613bc923976fb657b277c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/653ad286691eb3a87afecd5d82fb57fb931e7c0087a3e07358fc4d91241450c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/653ad286691eb3a87afecd5d82fb57fb931e7c0087a3e07358fc4d91241450c5/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/754f29b6870723d850df46b02b896181e6a4589d4c281a6def380fd04e7b7cd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/754f29b6870723d850df46b02b896181e6a4589d4c281a6def380fd04e7b7cd5/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/793602ed89a001b62c8b5158c1055fb18c460d287078f21ee9de3d461e41fda3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/793602ed89a001b62c8b5158c1055fb18c460d287078f21ee9de3d461e41fda3/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f9f5ec11211c230915c9646b71f79ebdac153a2ce637e63381703933c494c27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f9f5ec11211c230915c9646b71f79ebdac153a2ce637e63381703933c494c27/userdata/shm major:0 minor:450 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/868be30f7f75bfd2a455980daefa80e1d9f7a3232223d43f4c1a9e5a69003a7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/868be30f7f75bfd2a455980daefa80e1d9f7a3232223d43f4c1a9e5a69003a7f/userdata/shm major:0 minor:445 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88ebbdb1e42452ff11f3007689a4e05014309f480269dc9a94aff80063132f55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88ebbdb1e42452ff11f3007689a4e05014309f480269dc9a94aff80063132f55/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94dc94fee5363e04e224b6fe3ba7458e32191b1d97e69cbcda60ed9a6df736a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94dc94fee5363e04e224b6fe3ba7458e32191b1d97e69cbcda60ed9a6df736a6/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac1476918d6cf0940208a74aea0cbef9585642173751d9e7adbdf93d52fd02a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac1476918d6cf0940208a74aea0cbef9585642173751d9e7adbdf93d52fd02a1/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac349d5b6c64518f91c5c6b4eaf6c3862d9c749288e6e783c9c6e67adb7dc44e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac349d5b6c64518f91c5c6b4eaf6c3862d9c749288e6e783c9c6e67adb7dc44e/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca9c819ac2c4e0f7a68114381873732c967e7b6f45c794dd45031758fcd8b44d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca9c819ac2c4e0f7a68114381873732c967e7b6f45c794dd45031758fcd8b44d/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0b901aef19e9ae7ac32978455430ac3537e8c68317adfd6daefe5b0cf5c495b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0b901aef19e9ae7ac32978455430ac3537e8c68317adfd6daefe5b0cf5c495b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4b26dcce58bd3521ca66c870cdc6619c39ae14f7647ff9c1efffa33f9e476ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4b26dcce58bd3521ca66c870cdc6619c39ae14f7647ff9c1efffa33f9e476ac/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01b136c4-f8d9-4b45-9454-ab09db908f72/volumes/kubernetes.io~projected/kube-api-access-67pfj:{mountpoint:/var/lib/kubelet/pods/01b136c4-f8d9-4b45-9454-ab09db908f72/volumes/kubernetes.io~projected/kube-api-access-67pfj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02c1435f-0c51-4647-b1c2-0334a39a1af2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/02c1435f-0c51-4647-b1c2-0334a39a1af2/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10df746e-9937-45a2-ae81-c39ddc189ea6/volumes/kubernetes.io~projected/kube-api-access-m2j52:{mountpoint:/var/lib/kubelet/pods/10df746e-9937-45a2-ae81-c39ddc189ea6/volumes/kubernetes.io~projected/kube-api-access-m2j52 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de/volumes/kubernetes.io~projected/kube-api-access-w4fhz:{mountpoint:/var/lib/kubelet/pods/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de/volumes/kubernetes.io~projected/kube-api-access-w4fhz major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volumes/kubernetes.io~projected/kube-api-access-77j56:{mountpoint:/var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volumes/kubernetes.io~projected/kube-api-access-77j56 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~projected/kube-api-access-qq9tj:{mountpoint:/var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~projected/kube-api-access-qq9tj major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~projected/kube-api-access-kfmvs:{mountpoint:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~projected/kube-api-access-kfmvs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~projected/kube-api-access-smh4k:{mountpoint:/var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~projected/kube-api-access-smh4k major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a4f8473-1c87-463e-8338-ee1971bd0064/volumes/kubernetes.io~projected/kube-api-access-5ncr4:{mountpoint:/var/lib/kubelet/pods/5a4f8473-1c87-463e-8338-ee1971bd0064/volumes/kubernetes.io~projected/kube-api-access-5ncr4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~projected/kube-api-access-kvrsz:{mountpoint:/var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~projected/kube-api-access-kvrsz major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7736e260-9632-4568-a43c-33b5eecd6acb/volumes/kubernetes.io~projected/kube-api-access-rc6f4:{mountpoint:/var/lib/kubelet/pods/7736e260-9632-4568-a43c-33b5eecd6acb/volumes/kubernetes.io~projected/kube-api-access-rc6f4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d5c0784-95be-40af-98df-14e58bf411f0/volumes/kubernetes.io~projected/kube-api-access-vxbjn:{mountpoint:/var/lib/kubelet/pods/7d5c0784-95be-40af-98df-14e58bf411f0/volumes/kubernetes.io~projected/kube-api-access-vxbjn major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d5c0784-95be-40af-98df-14e58bf411f0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7d5c0784-95be-40af-98df-14e58bf411f0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/889ada6d-d4ee-4f5e-aa22-23637608940a/volumes/kubernetes.io~projected/kube-api-access-zc79x:{mountpoint:/var/lib/kubelet/pods/889ada6d-d4ee-4f5e-aa22-23637608940a/volumes/kubernetes.io~projected/kube-api-access-zc79x major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/889ada6d-d4ee-4f5e-aa22-23637608940a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/889ada6d-d4ee-4f5e-aa22-23637608940a/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a49fe757-54a2-4871-b450-d48e15571c9c/volumes/kubernetes.io~projected/kube-api-access-jrhsz:{mountpoint:/var/lib/kubelet/pods/a49fe757-54a2-4871-b450-d48e15571c9c/volumes/kubernetes.io~projected/kube-api-access-jrhsz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a49fe757-54a2-4871-b450-d48e15571c9c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a49fe757-54a2-4871-b450-d48e15571c9c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b10257bd-92be-4617-beab-f2270c5d396d/volumes/kubernetes.io~projected/kube-api-access-bc7xd:{mountpoint:/var/lib/kubelet/pods/b10257bd-92be-4617-beab-f2270c5d396d/volumes/kubernetes.io~projected/kube-api-access-bc7xd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b10257bd-92be-4617-beab-f2270c5d396d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b10257bd-92be-4617-beab-f2270c5d396d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/config-reloader/2 major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/prometheus/15 major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~empty-dir/config-out major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~projected/kube-api-access-sdvvz:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~projected/kube-api-access-sdvvz major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~projected/tls-assets major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/config major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/web-config major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~projected/kube-api-access-p9fld:{mountpoint:/var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~projected/kube-api-access-p9fld major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1e57895-4080-4a05-a5b0-b859cdfd96dc/volumes/kubernetes.io~projected/kube-api-access-4wln6:{mountpoint:/var/lib/kubelet/pods/e1e57895-4080-4a05-a5b0-b859cdfd96dc/volumes/kubernetes.io~projected/kube-api-access-4wln6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9/volumes/kubernetes.io~projected/kube-api-access-cb7tj:{mountpoint:/var/lib/kubelet/pods/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9/volumes/kubernetes.io~projected/kube-api-access-cb7tj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~projected/kube-api-access-wdmw5:{mountpoint:/var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~projected/kube-api-access-wdmw5 major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~projected/kube-api-access-tz6rw:{mountpoint:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~projected/kube-api-access-tz6rw major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/alertmanager/9 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/config-reloader/6 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~empty-dir/config-out major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~projected/kube-api-access-dw9kr:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~projected/kube-api-access-dw9kr major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~projected/tls-assets major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/config-volume major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/web-config major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~projected/kube-api-access-vpr2d:{mountpoint:/var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~projected/kube-api-access-vpr2d major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:267 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ea33383a005d99ab6606f8589aa12afdc8045b87744a1c45f8d6d15b75891faf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8e6daddbc758bcb297a547f97a813bb86ddbdbabe8517fd212c93fb8ee353462/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bd96b792c964ed5a0eaaf41824cf427e1e1362c4a3fbe449fd164636ae30dc87/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2f52b43d9a8076742d25806834439b24567386daed54aac3a87fbac71a04b1a1/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f336506edabbab79a3dc0a37b380fc30b9c2ccfe4500b139bf7c776406e1c402/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0005ae74997b7dca3d6a52a7195171ca0c508bf8a79a12f9ace8b95e74888ae1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/45322a74d78a43d0ffb3eb76dca20b419938382c2533494e0e363a6a0de09a4d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/190972eb5c84f4f7e3503e06176868fe46c388d9af1f00ac6b2c98056fdd6d80/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9e5b2f9d295546e9d423dd9e86b6ffe2a8d943f2cf19332e3f302f24a22b7e41/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/94022e185024dabc2fd9538bf9486df1edda1229244f39d96a2c59ef24d3076c/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/94f170faa641492e17b347b71ed13863e162237e01fa23c0ac89cc467697fb23/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/4524969a327235aedab5b4cba75f0452d4f018a0555010f17ac3c904108731ed/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/b5a59f3dcc7333b55570810c23f687af4cb4c6264f9576aa15707f1a7fe5da20/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/8d00d09aa4afa62232ddc5b7e737c05000d875509d60a476f56b52bd98f4b371/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7bf7b7c705bde899bd930ae62ea5a306f0759e7e78f91ce529fb87ed83549c35/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3d6a62ffa3ba0ade0940f3ec8e9d32f3d7bc105c07939f5e68916c9d2d9fcc75/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d86f577f629925456ace49fdc17ea18f68956f2a59aff4ba2f402b22685445c6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d16e317bfaaeee0bfe33b3e39c98e52ac4fcbd1832644a4cac26fb44bd4c6fd2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d995f8254e3078d368568d6db9d7e56f4dfdf28cb0d0916e2d23721aaa718ea8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c03468cd1017ce3fb861ed2f1b10e1292b364c0eb64f905b03518fa37ecb3ab9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/30246f79437503a59ad73d5518ac13bee7c8572a194bd1ada215f55b506748fd/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/6f9726844095c57f571a7628fed8f175bbedeab4d715767911d77861e5fd9673/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8f6447edf6dc30ba6a4532245815ddfb883b8fe72a0c35ce9769b9716afa4b25/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/82e4eacf4591c7ed60116f4ebbd8e869e6d36160578f7bae21f6ee6960b0c63a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ea795094daf1e6eb5a032d9594f17678c70859f2c412f78e0c4312f02641fadd/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/9dc32ffa6c786080a2c1d17caa4215e363390389602fe9067af9511c5f807676/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/27fad835a022bfdf4e2e1a42bd34947a65e4d12e7cb4051170950ef48e66fd3e/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8f6ea22e6691260e070ee8f9315ee8570b7cf53c31d6e751de6c27d22cd8f961/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/0f6a8b6eb2d543e3385204a271b88fd0e0fb47f1e876bd65c6ee60275d28f035/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/3d543f1aff2fa84e6574b8808652aa6c216023d65ce890291715cb591f3a4d9e/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/bb6ae2de1b47db43df7ad44cceaf15834cf14f614b397cc9641d2712346509d7/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/7421727ad4c58b76c74d1779b3d0c96c65484b7d65d3a6ed6e125c392dcb0cc1/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/8a56b31fcb5dc7b05257a9a38786e1f224ab6ccc5dbbfb3bb3ebc8ae52c7ef04/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/0566acb5e8c71ab3a543a0341f546ee19e187a149c617bbcd360e76a7de1c9ae/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5ee2116a7b29d3eef7f21bb2fc3ee95b952f5806a8fa81eae95066934e766c18/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/e0f4db4301f09d0032705b6a5674c7fbb6a7cd923bfb6aa867c960ed36311b80/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/34506cb6351a8aeec09e9b779fd57210ec01402f796d551a12e1c34ac3201682/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/e502da84a685a78b4280df654ac3674cfd568451189f14562b1f19c1c210d2af/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/cc966367280890c4cd111ab4dd340cbca7f4c45259092e307bb30328f2729453/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/a7ff042266fc041eec816663a20c48897d3eaff70a648b2129eefc60c2e2d264/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/5feff3ac6f77cfd8ccd6003b3fd0663828f8372295db9f0307d769399d14acbb/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/fc8e37d080004360de4797114e80b9b636b5c09716efbac65718d187f66a5d49/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/96873cf17e9a8da14c30d77f7acc07b1afe41b0e30a34beaf519de92f3381364/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/a6fa36f275b61d233a5093c379160ac6a1e9d724301ebba6e20a542fd1f995ea/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/9aa3ba8f6d8b165d49b44ed173786921944b5fbe633d6861c8ba416a82b8ac56/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/ce8c60d15184c01835b17a010668a7ab8ba25fd9173839bb16e9c6244cf34a33/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/94979ccd46df0101a78c1f62073b57d675f5b88600c85b8bef9422b289e14047/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/ba7934d9778a50bd22d49eee0b60fb64d1fe14179f858cbea936a980c38b105c/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/16efa24afe1ee63085f9bb5f4a4d0ab456276864d7f5619d6b6cf4ddcd6838b4/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/fbbd1b30aa5c7d7642efa6dd2ad7d64e1af1d28500b043b123388eeb2d94a4cb/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/e942b51aa650f22c96695889646b7bbe514a231dec00c9ddd7b90a0ccadd8ea7/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/424887650eb917744bbdb89ecae79124085b7b7803d4608928941ecbebcdbc58/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/739b5ca9e1aed05672fce7c19bda8d1b15e25a6a2c7c0f09582aa31094ea2acd/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/52cc658aece488646d4bbae9cdf8c1a3cecb380a94bcdfa432b653d659cfd759/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/032325bea01a387be3d02f8e4bc19f5d8d495c0d994cbc01c414f2d9a322b38f/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/0034619ad2592e7c9092967e9f19292b339a35645c72807ba1e7cb9d3bf7fe48/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/fce3d69b35d01ea4700e6efa9d7699b175904247123aee024c10342b70731e39/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/4d3ec7b61df6551dbe749421c96ffb268ec99e2cfd7497b44c7e967e2f591f58/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/38e9e8044bbec04d6e7793cb7583930e1953e09d39e6faeb00bdaab65543f1df/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/29f3f66d351dca0506878b5b19a13564f93ab21fcb90887ea9202ad8e76f879e/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/a57ac51652ad8a9162ab31137817ef8598a11875fc0bd0087237a93bdec316d4/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/9017844e6c114007d95d5e1399ab79ce54a56ce83a026b6e15df40c24db27acd/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/dbc1d57bb8693bed5a9eae79cb1610d0da1cbf4eca44c55c41fb1941f2e79f0b/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/ac7f312504d5fd88ca42287cf79c6150232b4b417589d76eab9eee3f406d5511/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/e6442332cc9bc4640c2da81cd01bee5f0be392db199b8b914c6cd9e0651ae3cd/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/cd5779d8dd558d283e969b75609b45d8d0682bb070861d8a144840d2c5bd2437/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/bc3c607985d6a44f04fe06735a9fc749b5d4d75c8c006aeffbcb801791202ab9/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/bd940edb4e6659f86ec79c724ec1c6f984f6030b4399c6edbfc2eec3dc4d8a0f/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/a5f6e46d7b22ee182f73bbd82eae34e95de644212a2b12dc8436b66fd9cc7921/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/666d233835e9af20cd4f5aa9f42f15f4141e68b0f4c91cf04cb07eec5b26fb92/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/c7dfc4d18802e74ddbad72858c3e5409de3641131efbfb1dbaf44eb70c2f7964/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/c1c6ea2a20455b11bb90992437cacff622e299b39b1f2a0b1c50437429e55f4f/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/a9118308fd67b492f7489fcdc4d98b0685ce2425104b63f8e27a9ee3f1d61a18/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4b21cd4ac9fe1c25dfaa4dccf016c0c39caefc18aa941a95eb726a61f216d4f6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/6bdc50a7b0d43186f150cba3737def743f50291ec44a94782df5bd9f777e5159/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/d9441402272b478038fef2fdc60e09c3fb1c4ad7433232e46c87d9be03ed5d1a/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/25cd184dcc1b861f1ac3ccffe3adb01b42be2593f59ef4e39368c368802cfb81/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/7dcc30577443a01cd123d915f7414edbbc00ace90e7654fa50bcbfa1a4d13eb8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/fcaaa86bfbe52739c5a03e2eb0fc387e6eed0acfa8a084fc2d899428dc2c95b7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4ff0c0e148c42b5cd0b351651125a13eb6beb87aa4a3fcf60a902307ed9db3c8/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/abc88e3ea2842a7a763ea23b292914394a18010946ecf921637706f3c784b40b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9452374ca39af6f3fd0400ccf857713737d2bb32d90d68f75c95b21042dabac7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b4fc7ec26a296f439784fcb0cc4ebe8714e42447f750ddfa4ca35c22c43bddd8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d91ef40a6e22225742389b8ffe1bcf1941a2ea83b25c996c2a113468952fc584/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e479ab2e83c193579e2b1c111afbb55f5424c1703753e8492f17cb968a1be1d9/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 21:15:32.450183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.449725 5931 manager.go:217] Machine: {Timestamp:2026-11-03 21:15:32.449233412 +0000 UTC m=+0.060081238 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200070 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c8162b6e8fefb291cb2ad7da3da7f SystemUUID:ec2c8162-b6e8-fefb-291c-b2ad7da3da7f BootID:2f33450a-8758-4c3e-8fca-e06c97bcc736 Filesystems:[{Device:/run/containers/storage/overlay-containers/38c585b403cc8fb42f8498fa7259ece6fb9a1137be5069db08c526f58f27ade8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2de3fa79e5c8b33275c84f5724425e8da7962e2d41a7f67406e766d06fc9c3d9/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:319 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~projected/kube-api-access-smh4k DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/10df746e-9937-45a2-ae81-c39ddc189ea6/volumes/kubernetes.io~projected/kube-api-access-m2j52 DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/889ada6d-d4ee-4f5e-aa22-23637608940a/volumes/kubernetes.io~projected/kube-api-access-zc79x DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:314 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:320 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:319 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:430 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4b26dcce58bd3521ca66c870cdc6619c39ae14f7647ff9c1efffa33f9e476ac/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:430 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac1476918d6cf0940208a74aea0cbef9585642173751d9e7adbdf93d52fd02a1/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0b901aef19e9ae7ac32978455430ac3537e8c68317adfd6daefe5b0cf5c495b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:316 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:437 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/653ad286691eb3a87afecd5d82fb57fb931e7c0087a3e07358fc4d91241450c5/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d5c0784-95be-40af-98df-14e58bf411f0/volumes/kubernetes.io~projected/kube-api-access-vxbjn DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~projected/kube-api-access-p9fld DeviceMajor:0 DeviceMinor:272 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:435 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a49fe757-54a2-4871-b450-d48e15571c9c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:443 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:319 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de/volumes/kubernetes.io~projected/kube-api-access-w4fhz DeviceMajor:0 DeviceMinor:449 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/754f29b6870723d850df46b02b896181e6a4589d4c281a6def380fd04e7b7cd5/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:318 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:436 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:316 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:434 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88ebbdb1e42452ff11f3007689a4e05014309f480269dc9a94aff80063132f55/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~projected/kube-api-access-dw9kr DeviceMajor:0 DeviceMinor:326 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/153b1abd7c935b1a467680a61676c9e0d4c288e22d517c904a18cea07a2edacf/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f0f1d254373a18301419cbd223a7bc66a5878b43e4cd615e1bee8a51f8644be/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~projected/kube-api-access-qq9tj DeviceMajor:0 DeviceMinor:241 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:429 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~projected/kube-api-access-vpr2d DeviceMajor:0 DeviceMinor:273 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:470 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:430 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a4f8473-1c87-463e-8338-ee1971bd0064/volumes/kubernetes.io~projected/kube-api-access-5ncr4 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b10257bd-92be-4617-beab-f2270c5d396d/volumes/kubernetes.io~projected/kube-api-access-bc7xd DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:332 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:313 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~projected/kube-api-access-tz6rw DeviceMajor:0 DeviceMinor:354 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~projected/kube-api-access-sdvvz DeviceMajor:0 DeviceMinor:438 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca9c819ac2c4e0f7a68114381873732c967e7b6f45c794dd45031758fcd8b44d/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:428 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e7e0019da9070b86d795993d42d59571cf48906339613bc923976fb657b277c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:267 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:315 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:317 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:331 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:432 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:433 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/01b136c4-f8d9-4b45-9454-ab09db908f72/volumes/kubernetes.io~projected/kube-api-access-67pfj DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:430 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e68f3a539d2936c561295045e6f0c76e536ccf481c2fe8abfead99c1aead68f/userdata/shm DeviceMajor:0 DeviceMinor:447 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9/volumes/kubernetes.io~projected/kube-api-access-cb7tj DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4f4cad3e-0d13-4459-b970-79675d07443d/volumes/kubernetes.io~projected/kube-api-access-77j56 DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~projected/kube-api-access-kvrsz DeviceMajor:0 DeviceMinor:265 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~projected/kube-api-access-kfmvs DeviceMajor:0 DeviceMinor:334 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/793602ed89a001b62c8b5158c1055fb18c460d287078f21ee9de3d461e41fda3/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:424 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:466 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/02c1435f-0c51-4647-b1c2-0334a39a1af2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/59fd2087-a50b-4e0a-8439-379d3094ee59/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fffd8c3e-675e-45b0-b22f-445e5a92bbfd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:268 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:321 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:330 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:259 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac349d5b6c64518f91c5c6b4eaf6c3862d9c749288e6e783c9c6e67adb7dc44e/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94dc94fee5363e04e224b6fe3ba7458e32191b1d97e69cbcda60ed9a6df736a6/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/889ada6d-d4ee-4f5e-aa22-23637608940a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/868be30f7f75bfd2a455980daefa80e1d9f7a3232223d43f4c1a9e5a69003a7f/userdata/shm DeviceMajor:0 DeviceMinor:445 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:319 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:431 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:444 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes/kubernetes.io~projected/kube-api-access-wdmw5 DeviceMajor:0 DeviceMinor:471 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d5c0784-95be-40af-98df-14e58bf411f0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:352 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1e57895-4080-4a05-a5b0-b859cdfd96dc/volumes/kubernetes.io~projected/kube-api-access-4wln6 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51c80b41da04d7ba26386c909d3b07e93a25cc458b0d82ffc9cde5d8efa04452/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:240 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:353 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f9f5ec11211c230915c9646b71f79ebdac153a2ce637e63381703933c494c27/userdata/shm DeviceMajor:0 DeviceMinor:450 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7736e260-9632-4568-a43c-33b5eecd6acb/volumes/kubernetes.io~projected/kube-api-access-rc6f4 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b10257bd-92be-4617-beab-f2270c5d396d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/befe88bf-8d8b-47b8-a244-06524e4c3e28/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:266 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58d5c977-b290-45f5-a219-8f2a98eab0f6/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:328 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f070dd5b-30ed-4531-8cf6-196779398aad/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:351 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a49fe757-54a2-4871-b450-d48e15571c9c/volumes/kubernetes.io~projected/kube-api-access-jrhsz DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1e68f3a539d2936 MacAddress:4a:63:28:6d:af:d5 Speed:10000 Mtu:8901} {Name:2de3fa79e5c8b33 MacAddress:62:e3:09:7d:2a:37 Speed:10000 Mtu:8901} {Name:47178319b642438 MacAddress:6a:1d:3f:68:52:0a Speed:10000 Mtu:8901} {Name:5e7e0019da9070b MacAddress:a2:6d:f3:1c:b4:9c Speed:10000 Mtu:8901} {Name:754f29b6870723d MacAddress:02:bb:02:f8:97:20 Speed:10000 Mtu:8901} {Name:793602ed89a001b MacAddress:d6:e3:7b:5e:44:0c Speed:10000 Mtu:8901} {Name:7f9f5ec11211c23 MacAddress:86:a9:9e:00:87:3a Speed:10000 Mtu:8901} {Name:868be30f7f75bfd MacAddress:02:a1:44:76:38:7c Speed:10000 Mtu:8901} {Name:97faeb06d3d26c6 MacAddress:e2:46:9e:70:25:a8 Speed:10000 Mtu:8901} {Name:9c977e420ca9750 MacAddress:46:90:92:78:e9:e8 Speed:10000 Mtu:8901} {Name:ac349d5b6c64518 MacAddress:ee:e7:e6:1a:36:e4 Speed:10000 Mtu:8901} {Name:adac3c9dac7b0bb MacAddress:12:cd:9b:f6:1f:17 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:0d:71:0f:3d:03 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:0e:36:a4:dc:17 Speed:0 Mtu:8901} {Name:ca9c819ac2c4e0f MacAddress:3a:68:5d:28:13:67 Speed:10000 Mtu:8901} {Name:d4b26dcce58bd35 MacAddress:ee:66:d9:87:5c:01 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:0d:71:0f:3d:03 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:48:2f:9f:66:e8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:47:b7:55:ce:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 21:15:32.450183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450185 5931 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 21:15:32.450657 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450245 5931 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 21:15:32.450657 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450547 5931 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 21:15:32.450775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450569 5931 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-179.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 21:15:32.450845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450779 5931 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 21:15:32.450845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450793 5931 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 21:15:32.450845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450808 5931 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:15:32.450845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450836 5931 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:15:32.450987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.450972 5931 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 21:15:32.451164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.451150 5931 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 21:15:32.451290 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.451278 5931 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 21:15:32.451342 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.451326 5931 kubelet.go:493] "Attempting to sync node with API server" Nov 03 21:15:32.451342 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.451340 5931 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 21:15:32.451433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.451355 5931 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 21:15:32.451433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.451368 5931 kubelet.go:405] "Adding apiserver pod source" Nov 03 21:15:32.451433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.451382 5931 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 21:15:32.452120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.452104 5931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 21:15:32.452120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.452118 5931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:15:32.452717 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.452697 5931 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 21:15:32.455421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455408 5931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 21:15:32.455476 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455422 5931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:15:32.455519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455510 5931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 21:15:32.455725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455714 5931 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 21:15:32.455763 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455744 5931 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 21:15:32.455838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455829 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 21:15:32.455866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455844 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 21:15:32.455866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455850 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 21:15:32.455866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455856 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 21:15:32.455866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455861 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 21:15:32.455866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455867 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455873 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455878 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455885 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455890 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455899 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455908 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455934 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 21:15:32.455995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.455940 5931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 21:15:32.459248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.459235 5931 server.go:1263] "Started kubelet" Nov 03 21:15:32.459309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.459287 5931 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 21:15:32.459729 ip-10-0-128-179 systemd[1]: Started Kubernetes Kubelet. Nov 03 21:15:32.460446 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.459783 5931 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 21:15:32.460446 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.459857 5931 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 21:15:32.461016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.460995 5931 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 21:15:32.465400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465384 5931 server.go:317] "Adding debug handlers to kubelet server" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465603 5931 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465642 5931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465756 5931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:21 +0000 UTC" deadline="2028-04-26 11:03:03.831558724 +0000 UTC" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465778 5931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12949h47m31.365783758s" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465810 5931 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465816 5931 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.465842 5931 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:15:32.466218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.465926 5931 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 21:15:32.469046 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.468996 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-grpc-tls" seLinuxMountContext="" Nov 03 21:15:32.469129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469048 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01b136c4-f8d9-4b45-9454-ab09db908f72" volumeName="kubernetes.io/configmap/01b136c4-f8d9-4b45-9454-ab09db908f72-serviceca" seLinuxMountContext="" Nov 03 21:15:32.469129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469064 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f070dd5b-30ed-4531-8cf6-196779398aad" volumeName="kubernetes.io/empty-dir/f070dd5b-30ed-4531-8cf6-196779398aad-audit-log" seLinuxMountContext="" Nov 03 21:15:32.469129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469076 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f070dd5b-30ed-4531-8cf6-196779398aad" volumeName="kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 21:15:32.469129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469097 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f4cad3e-0d13-4459-b970-79675d07443d" volumeName="kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-script-lib" seLinuxMountContext="" Nov 03 21:15:32.469129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469108 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a4f8473-1c87-463e-8338-ee1971bd0064" volumeName="kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-cni-binary-copy" seLinuxMountContext="" Nov 03 21:15:32.469129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469121 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7459ce-aa09-4c74-b5ef-8424cb8d24b5" volumeName="kubernetes.io/projected/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-kube-api-access-kvrsz" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469132 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469147 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469159 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50541dcd-b16b-493b-9675-0f232b6abb4e" volumeName="kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469170 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59fd2087-a50b-4e0a-8439-379d3094ee59" volumeName="kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-tmp" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469181 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469192 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469203 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469216 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469228 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469239 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469255 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7736e260-9632-4568-a43c-33b5eecd6acb" volumeName="kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469266 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5c0784-95be-40af-98df-14e58bf411f0" volumeName="kubernetes.io/secret/7d5c0784-95be-40af-98df-14e58bf411f0-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469277 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469291 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469302 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b10257bd-92be-4617-beab-f2270c5d396d" volumeName="kubernetes.io/configmap/b10257bd-92be-4617-beab-f2270c5d396d-config-volume" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469313 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469323 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b10257bd-92be-4617-beab-f2270c5d396d" volumeName="kubernetes.io/secret/b10257bd-92be-4617-beab-f2270c5d396d-metrics-tls" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469336 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469347 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469360 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59fd2087-a50b-4e0a-8439-379d3094ee59" volumeName="kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-tuned" seLinuxMountContext="" Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469371 5931 factory.go:153] Registering CRI-O factory Nov 03 21:15:32.469395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469389 5931 factory.go:223] Registration of the crio container factory successfully Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469439 5931 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469372 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8acde6b-bb34-4c16-9afe-aa4c21f59dd6" volumeName="kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469465 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f070dd5b-30ed-4531-8cf6-196779398aad" volumeName="kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469479 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469491 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f4cad3e-0d13-4459-b970-79675d07443d" volumeName="kubernetes.io/secret/4f4cad3e-0d13-4459-b970-79675d07443d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469503 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a4f8473-1c87-463e-8338-ee1971bd0064" volumeName="kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-daemon-config" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469516 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469447 5931 factory.go:55] Registering systemd factory Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469527 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469534 5931 factory.go:223] Registration of the systemd container factory successfully Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469539 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="889ada6d-d4ee-4f5e-aa22-23637608940a" volumeName="kubernetes.io/secret/889ada6d-d4ee-4f5e-aa22-23637608940a-cert" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469555 5931 factory.go:103] Registering Raw factory Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469569 5931 manager.go:1199] Started watching for new ooms in manager Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469900 5931 manager.go:319] Starting recovery of all containers Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.469550 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470055 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/projected/58d5c977-b290-45f5-a219-8f2a98eab0f6-kube-api-access-kfmvs" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470079 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470093 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10df746e-9937-45a2-ae81-c39ddc189ea6" volumeName="kubernetes.io/projected/10df746e-9937-45a2-ae81-c39ddc189ea6-kube-api-access-m2j52" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470106 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50541dcd-b16b-493b-9675-0f232b6abb4e" volumeName="kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470140 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5c0784-95be-40af-98df-14e58bf411f0" volumeName="kubernetes.io/configmap/7d5c0784-95be-40af-98df-14e58bf411f0-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470156 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470170 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50541dcd-b16b-493b-9675-0f232b6abb4e" volumeName="kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470183 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a4f8473-1c87-463e-8338-ee1971bd0064" volumeName="kubernetes.io/projected/5a4f8473-1c87-463e-8338-ee1971bd0064-kube-api-access-5ncr4" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470196 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b10257bd-92be-4617-beab-f2270c5d396d" volumeName="kubernetes.io/empty-dir/b10257bd-92be-4617-beab-f2270c5d396d-tmp-dir" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470209 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befe88bf-8d8b-47b8-a244-06524e4c3e28" volumeName="kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470221 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f070dd5b-30ed-4531-8cf6-196779398aad" volumeName="kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470233 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" volumeName="kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-tls" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470246 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470259 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e346f0cc-7ed6-42f5-a4f9-1c16320cefd9" volumeName="kubernetes.io/projected/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kube-api-access-cb7tj" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470271 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470284 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de" volumeName="kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470298 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/empty-dir/58d5c977-b290-45f5-a219-8f2a98eab0f6-tmp" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470311 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470325 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f070dd5b-30ed-4531-8cf6-196779398aad" volumeName="kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470337 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" volumeName="kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-metrics-client-ca" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470363 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7736e260-9632-4568-a43c-33b5eecd6acb" volumeName="kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-binary-copy" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470375 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f4cad3e-0d13-4459-b970-79675d07443d" volumeName="kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-config" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470387 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470399 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470413 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a49fe757-54a2-4871-b450-d48e15571c9c" volumeName="kubernetes.io/projected/a49fe757-54a2-4871-b450-d48e15571c9c-kube-api-access-jrhsz" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470429 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7736e260-9632-4568-a43c-33b5eecd6acb" volumeName="kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470443 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7736e260-9632-4568-a43c-33b5eecd6acb" volumeName="kubernetes.io/projected/7736e260-9632-4568-a43c-33b5eecd6acb-kube-api-access-rc6f4" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470455 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470488 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befe88bf-8d8b-47b8-a244-06524e4c3e28" volumeName="kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-metrics-client-ca" seLinuxMountContext="" Nov 03 21:15:32.470480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470503 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470516 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470529 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470543 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50541dcd-b16b-493b-9675-0f232b6abb4e" volumeName="kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470558 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befe88bf-8d8b-47b8-a244-06524e4c3e28" volumeName="kubernetes.io/empty-dir/befe88bf-8d8b-47b8-a244-06524e4c3e28-volume-directive-shadow" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470571 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befe88bf-8d8b-47b8-a244-06524e4c3e28" volumeName="kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470583 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1e57895-4080-4a05-a5b0-b859cdfd96dc" volumeName="kubernetes.io/projected/e1e57895-4080-4a05-a5b0-b859cdfd96dc-kube-api-access-4wln6" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470597 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470626 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50541dcd-b16b-493b-9675-0f232b6abb4e" volumeName="kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470639 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7459ce-aa09-4c74-b5ef-8424cb8d24b5" volumeName="kubernetes.io/configmap/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-metrics-client-ca" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470652 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02c1435f-0c51-4647-b1c2-0334a39a1af2" volumeName="kubernetes.io/configmap/02c1435f-0c51-4647-b1c2-0334a39a1af2-konnectivity-ca" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470665 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470678 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470691 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f070dd5b-30ed-4531-8cf6-196779398aad" volumeName="kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-client-ca-bundle" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470702 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470715 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470727 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7459ce-aa09-4c74-b5ef-8424cb8d24b5" volumeName="kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470740 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1e57895-4080-4a05-a5b0-b859cdfd96dc" volumeName="kubernetes.io/configmap/e1e57895-4080-4a05-a5b0-b859cdfd96dc-iptables-alerter-script" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470753 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470766 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470780 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50541dcd-b16b-493b-9675-0f232b6abb4e" volumeName="kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470791 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01b136c4-f8d9-4b45-9454-ab09db908f72" volumeName="kubernetes.io/projected/01b136c4-f8d9-4b45-9454-ab09db908f72-kube-api-access-67pfj" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470804 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10df746e-9937-45a2-ae81-c39ddc189ea6" volumeName="kubernetes.io/empty-dir/10df746e-9937-45a2-ae81-c39ddc189ea6-tmp-dir" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470817 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470830 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470844 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" volumeName="kubernetes.io/empty-dir/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-textfile" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470856 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befe88bf-8d8b-47b8-a244-06524e4c3e28" volumeName="kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470868 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f070dd5b-30ed-4531-8cf6-196779398aad" volumeName="kubernetes.io/projected/f070dd5b-30ed-4531-8cf6-196779398aad-kube-api-access-tz6rw" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470880 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470892 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f4cad3e-0d13-4459-b970-79675d07443d" volumeName="kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-env-overrides" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470904 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="889ada6d-d4ee-4f5e-aa22-23637608940a" volumeName="kubernetes.io/projected/889ada6d-d4ee-4f5e-aa22-23637608940a-kube-api-access-zc79x" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470918 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470930 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470941 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" volumeName="kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470955 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f4cad3e-0d13-4459-b970-79675d07443d" volumeName="kubernetes.io/projected/4f4cad3e-0d13-4459-b970-79675d07443d-kube-api-access-77j56" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470968 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b10257bd-92be-4617-beab-f2270c5d396d" volumeName="kubernetes.io/projected/b10257bd-92be-4617-beab-f2270c5d396d-kube-api-access-bc7xd" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470982 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.470996 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471007 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59fd2087-a50b-4e0a-8439-379d3094ee59" volumeName="kubernetes.io/projected/59fd2087-a50b-4e0a-8439-379d3094ee59-kube-api-access-smh4k" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471019 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7459ce-aa09-4c74-b5ef-8424cb8d24b5" volumeName="kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471031 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a49fe757-54a2-4871-b450-d48e15571c9c" volumeName="kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471044 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471056 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" volumeName="kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471069 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec556062-fda8-4a75-aac7-fb519d35285d" volumeName="kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471081 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9816356-82fa-42f8-b5b4-b5fd84a32042" volumeName="kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471093 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" volumeName="kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471106 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5c0784-95be-40af-98df-14e58bf411f0" volumeName="kubernetes.io/projected/7d5c0784-95be-40af-98df-14e58bf411f0-kube-api-access-vxbjn" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471136 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" volumeName="kubernetes.io/projected/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-kube-api-access-vpr2d" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471148 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5c0784-95be-40af-98df-14e58bf411f0" volumeName="kubernetes.io/empty-dir/7d5c0784-95be-40af-98df-14e58bf411f0-data-volume" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471161 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02c1435f-0c51-4647-b1c2-0334a39a1af2" volumeName="kubernetes.io/secret/02c1435f-0c51-4647-b1c2-0334a39a1af2-agent-certs" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471173 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50541dcd-b16b-493b-9675-0f232b6abb4e" volumeName="kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471185 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befe88bf-8d8b-47b8-a244-06524e4c3e28" volumeName="kubernetes.io/projected/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-api-access-p9fld" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471200 5931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d5c977-b290-45f5-a219-8f2a98eab0f6" volumeName="kubernetes.io/configmap/58d5c977-b290-45f5-a219-8f2a98eab0f6-metrics-client-ca" seLinuxMountContext="" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471212 5931 reconstruct.go:97] "Volume reconstruction finished" Nov 03 21:15:32.472454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.471219 5931 reconciler.go:29] "Reconciler: start to sync state" Nov 03 21:15:32.496510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.496469 5931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 21:15:32.497981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.497959 5931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 21:15:32.497981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.497982 5931 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 21:15:32.498125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.498004 5931 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 21:15:32.498125 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.498043 5931 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 21:15:32.502260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.502227 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:15:32.504236 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.504211 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:15:32.516752 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.516727 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd" exitCode=0 Nov 03 21:15:32.522546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.522518 5931 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="31de5d3a0376d4b66f3510706996a37aa356a4bd858f5f4346df57db03aa5a96" exitCode=0 Nov 03 21:15:32.522546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.522545 5931 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="afdf65282ae5675c5035d67cd1ebddf00f398fd6535aa34033961033b34fac6c" exitCode=0 Nov 03 21:15:32.522730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.522554 5931 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="093bb622a293fda525d7ada3078750df1cff6b222b18cfc09c4d7692741511cf" exitCode=0 Nov 03 21:15:32.522730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.522563 5931 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="39a23047b5528a36aab4da63d5642cf013dd320b85024f9c3a97a0c0ee327f78" exitCode=0 Nov 03 21:15:32.522730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.522570 5931 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="b36323632826442ef943c0257a6ed2e953ba5a9d7d5461471eee7e851448ccda" exitCode=0 Nov 03 21:15:32.522730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.522578 5931 generic.go:356] "Generic (PLEG): container finished" podID="7736e260-9632-4568-a43c-33b5eecd6acb" containerID="13cd182d80f098e3652b88beefd16144ecec709d3f6a889f86a64192816c7983" exitCode=0 Nov 03 21:15:32.524475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.524454 5931 generic.go:356] "Generic (PLEG): container finished" podID="c63ea962a14cc8c251237d3fd81ff3a9" containerID="ccf5087107f5f7564802d8e275656cebe9971b2f2edf267e135a619b2e14623d" exitCode=0 Nov 03 21:15:32.528526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.528505 5931 generic.go:356] "Generic (PLEG): container finished" podID="fffd8c3e-675e-45b0-b22f-445e5a92bbfd" containerID="97912fdef66d9d68f9492ddd36b596368caf8a0f04417617657ec921324a727c" exitCode=0 Nov 03 21:15:32.538512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.538492 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:15:32.538870 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.538845 5931 generic.go:356] "Generic (PLEG): container finished" podID="4f4cad3e-0d13-4459-b970-79675d07443d" containerID="7107698647ae1637a61db56dd070e6dd04eccc291eb636d6ebb7ad4e99392dfd" exitCode=1 Nov 03 21:15:32.550738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.550717 5931 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="1e0348bb86e8b11e81b7d2aab96b40e40ad38da22be1d0969cd611165dee501e" exitCode=0 Nov 03 21:15:32.565999 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.565977 5931 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:15:32.590939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.590922 5931 manager.go:324] Recovery completed Nov 03 21:15:32.598094 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.598074 5931 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 21:15:32.611838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.611822 5931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:15:32.613724 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.613708 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:15:32.613813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.613737 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:15:32.613813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.613749 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:15:32.618121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618097 5931 cpu_manager.go:226] "Starting" policy="none" Nov 03 21:15:32.618121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618119 5931 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 21:15:32.618228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618124 5931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:15:32.618228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618139 5931 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 21:15:32.618322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618285 5931 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 21:15:32.618322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618296 5931 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 21:15:32.618322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618313 5931 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 21:15:32.618322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618319 5931 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 21:15:32.618322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618324 5931 policy_none.go:50] "Start" Nov 03 21:15:32.618456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618362 5931 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 21:15:32.618456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.618380 5931 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 21:15:32.619871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.619845 5931 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 21:15:32.619871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.619870 5931 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 21:15:32.619997 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.619881 5931 policy_none.go:44] "Start" Nov 03 21:15:32.620465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.620447 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:15:32.620535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.620477 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:15:32.620535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.620490 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:15:32.624660 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.624635 5931 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 21:15:32.628420 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.628405 5931 manager.go:341] "Starting Device Plugin manager" Nov 03 21:15:32.628490 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.628428 5931 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 21:15:32.628490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.628437 5931 server.go:87] "Starting device plugin registration server" Nov 03 21:15:32.628765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.628752 5931 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 21:15:32.628800 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.628768 5931 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 21:15:32.628885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.628858 5931 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 21:15:32.628995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.628957 5931 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 21:15:32.629081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.629007 5931 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 21:15:32.629081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.629014 5931 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 21:15:32.629157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.629065 5931 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:15:32.628991475Z","UUID":"c791e534-9c5c-4206-9f99-3ac8da728e08","Handler":null,"Name":"","Endpoint":""} Nov 03 21:15:32.630563 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.630544 5931 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 21:15:32.630665 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.630596 5931 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-179.us-west-1.compute.internal\" not found" Nov 03 21:15:32.631488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.631471 5931 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 21:15:32.631557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.631497 5931 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 21:15:32.660062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.660040 5931 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:15:32.660578 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.660564 5931 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:15:32.660824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.660809 5931 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 21:15:32.660959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.660947 5931 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:15:32.729226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.729204 5931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:15:32.730231 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.730216 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:15:32.730319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.730249 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:15:32.730319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.730264 5931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 21:15:32.730404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.730343 5931 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.759010 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.758989 5931 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:15:32.798713 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.798656 5931 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal"] Nov 03 21:15:32.799348 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.799315 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"c63ea962a14cc8c251237d3fd81ff3a9","Type":"ContainerStarted","Data":"c66f79e2b1f12490f94dd6acfc695247002abe83e4ba13307f916c4a9e1b0e9c"} Nov 03 21:15:32.799431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.799355 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"c63ea962a14cc8c251237d3fd81ff3a9","Type":"ContainerDied","Data":"ccf5087107f5f7564802d8e275656cebe9971b2f2edf267e135a619b2e14623d"} Nov 03 21:15:32.799431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.799369 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"c63ea962a14cc8c251237d3fd81ff3a9","Type":"ContainerStarted","Data":"4feffef87284000a78b16cff58a11a46a830b6b7a797fcf5049347a59535fa4c"} Nov 03 21:15:32.799523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.799448 5931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.799523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.799455 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"1745eaf53680bc2ae777ad5a0ee0f5c6","Type":"ContainerStarted","Data":"3850cf730c19090b6dda10f6a664ac307d54fcf6d556fb578329272a5ab0cf7f"} Nov 03 21:15:32.799523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.799468 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" event={"ID":"1745eaf53680bc2ae777ad5a0ee0f5c6","Type":"ContainerStarted","Data":"bb5788e7d40a41ad9b0ada239587fe20270c91a42ccb3d820554f93006540722"} Nov 03 21:15:32.799523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.799486 5931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.858839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.858765 5931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:15:32.858839 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.858835 5931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.861716 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.861694 5931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:15:32.861777 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:32.861742 5931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.864887 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.864871 5931 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.864980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.864922 5931 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.962126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.962099 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1745eaf53680bc2ae777ad5a0ee0f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"1745eaf53680bc2ae777ad5a0ee0f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.962126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.962131 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:32.962316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:32.962148 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.063133 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.063108 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1745eaf53680bc2ae777ad5a0ee0f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"1745eaf53680bc2ae777ad5a0ee0f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.063133 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.063136 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.063297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.063155 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.063297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.063197 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.063297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.063213 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1745eaf53680bc2ae777ad5a0ee0f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"1745eaf53680bc2ae777ad5a0ee0f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.063297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.063226 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c63ea962a14cc8c251237d3fd81ff3a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" (UID: \"c63ea962a14cc8c251237d3fd81ff3a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.452262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.452230 5931 apiserver.go:52] "Watching apiserver" Nov 03 21:15:33.555665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.555636 5931 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 21:15:33.556298 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.556268 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thpr2","openshift-insights/insights-runtime-extractor-xcf7l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb","kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t","openshift-multus/multus-additional-cni-plugins-qjndk","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-nkqjv","openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb","openshift-ovn-kubernetes/ovnkube-node-x8bjk","openshift-cluster-node-tuning-operator/tuned-lwqh9","openshift-console/console-9797677c8-8gnvt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal","openshift-multus/multus-5wwt4","kube-system/konnectivity-agent-6ljk7","openshift-console/console-789d48d68-988j4","openshift-dns/dns-default-9srkw","openshift-image-registry/node-ca-lrwcd","openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh","openshift-monitoring/metrics-server-ff498dd48-lmxq5","openshift-monitoring/node-exporter-tbv9l","openshift-network-diagnostics/network-check-target-2xkcf","openshift-network-operator/iptables-alerter-jqr6h","kube-system/global-pull-secret-syncer-9vfbk","openshift-dns/node-resolver-rvzp6"] Nov 03 21:15:33.575854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.575830 5931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.576006 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.575994 5931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.609894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.609866 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 21:15:33.610180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610162 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.610467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610434 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 21:15:33.610467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610455 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbwk2\"" Nov 03 21:15:33.610647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610445 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 21:15:33.610647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610499 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cldzt\"" Nov 03 21:15:33.610647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610547 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.610647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610554 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzfft\"" Nov 03 21:15:33.610647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610638 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\"" Nov 03 21:15:33.610907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610893 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 21:15:33.610972 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610960 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 21:15:33.611024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.610965 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 21:15:33.611072 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611037 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.611255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611241 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.611454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611432 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.611454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611441 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.611592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611461 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.611592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611575 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.611592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611586 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 21:15:33.611736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611585 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxfhg\"" Nov 03 21:15:33.611736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611579 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lzq4j\"" Nov 03 21:15:33.611736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611664 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.611844 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611834 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.611896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611849 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:15:33.611896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611884 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611904 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611924 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611884 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611941 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611937 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611957 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4f9ng\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611965 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 21:15:33.611983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611965 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 21:15:33.612284 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.611990 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 21:15:33.612685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.612668 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.612685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.612679 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trdz7\"" Nov 03 21:15:33.612810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.612689 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 21:15:33.612810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.612706 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbt6z\"" Nov 03 21:15:33.612810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.612735 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4rzx\"" Nov 03 21:15:33.612907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.612814 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.612907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.612866 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 21:15:33.666859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.666826 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-sys\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.666979 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.666870 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.666979 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.666897 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-textfile\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.666979 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.666952 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-etc-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.667126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.666990 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-node-log\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.667126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667015 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-textfile\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.667126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667020 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.667126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667095 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.667300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667129 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.667300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667149 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.667300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667228 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.667300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667254 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-netns\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.667300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667279 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-systemd\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667309 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-tuned\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667352 5931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667350 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667394 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-log-socket\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:33.667403 5931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667409 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-tuned\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667418 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667460 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-multus-certs\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667486 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pfj\" (UniqueName: \"kubernetes.io/projected/01b136c4-f8d9-4b45-9454-ab09db908f72-kube-api-access-67pfj\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667490 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.667521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667508 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmvs\" (UniqueName: \"kubernetes.io/projected/58d5c977-b290-45f5-a219-8f2a98eab0f6-kube-api-access-kfmvs\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667538 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667556 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667573 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-bin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667652 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667676 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667757 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-tmp\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667782 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-netns\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667807 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d5c977-b290-45f5-a219-8f2a98eab0f6-tmp\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667819 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59fd2087-a50b-4e0a-8439-379d3094ee59-tmp\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667833 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrsz\" (UniqueName: \"kubernetes.io/projected/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-kube-api-access-kvrsz\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667894 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58d5c977-b290-45f5-a219-8f2a98eab0f6-tmp\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.667974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.667940 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-systemd-units\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.668346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668001 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-system-cni-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.668346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668028 5931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:15:33.668346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668061 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-wtmp\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.668346 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:33.668089 5931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-179.us-west-1.compute.internal" Nov 03 21:15:33.668346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668108 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-sys-fs\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.668586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668344 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-script-lib\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.668586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668434 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fld\" (UniqueName: \"kubernetes.io/projected/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-api-access-p9fld\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.668586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668470 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-dbus\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.668586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668497 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-daemon-config\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.668586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668524 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.668586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668580 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-k8s-cni-cncf-io\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.668586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668581 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-script-lib\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.668828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668696 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-conf-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.668828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668735 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-daemon-config\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.668828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668733 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.668828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668800 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b10257bd-92be-4617-beab-f2270c5d396d-metrics-tls\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.668946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668842 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.668946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668869 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.668946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668909 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6f4\" (UniqueName: \"kubernetes.io/projected/7736e260-9632-4568-a43c-33b5eecd6acb-kube-api-access-rc6f4\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.668946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668937 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.669076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668963 5931 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 21:15:33.669076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668970 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.669076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.668998 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-systemd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.669076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669023 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10df746e-9937-45a2-ae81-c39ddc189ea6-hosts-file\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:15:33.669076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669050 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncr4\" (UniqueName: \"kubernetes.io/projected/5a4f8473-1c87-463e-8338-ee1971bd0064-kube-api-access-5ncr4\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.669076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669072 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.669256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669074 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02c1435f-0c51-4647-b1c2-0334a39a1af2-konnectivity-ca\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:15:33.669256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669107 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.669256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669134 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-host\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.669256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669161 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b10257bd-92be-4617-beab-f2270c5d396d-config-volume\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.669256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669204 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5c0784-95be-40af-98df-14e58bf411f0-data-volume\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.669256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669224 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02c1435f-0c51-4647-b1c2-0334a39a1af2-konnectivity-ca\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:15:33.669256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669228 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669285 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpr2d\" (UniqueName: \"kubernetes.io/projected/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-kube-api-access-vpr2d\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669301 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5c0784-95be-40af-98df-14e58bf411f0-data-volume\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669317 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-kubelet-config\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669336 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669351 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-cnibin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669365 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669379 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-sys\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669396 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-kubernetes\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.669463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669420 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-kubelet\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669543 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669562 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669574 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669630 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-bin\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669664 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-socket-dir-parent\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669693 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10257bd-92be-4617-beab-f2270c5d396d-tmp-dir\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669710 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669741 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.669812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669772 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10257bd-92be-4617-beab-f2270c5d396d-tmp-dir\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669845 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-grpc-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669898 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669958 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-original-pull-secret\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.669986 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d5c977-b290-45f5-a219-8f2a98eab0f6-metrics-client-ca\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670015 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-slash\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670034 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-ovn\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670051 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-env-overrides\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670067 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wln6\" (UniqueName: \"kubernetes.io/projected/e1e57895-4080-4a05-a5b0-b859cdfd96dc-kube-api-access-4wln6\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670071 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49fe757-54a2-4871-b450-d48e15571c9c-metrics-certs\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670082 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-metrics-client-ca\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.670113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670096 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-conf\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670119 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7xd\" (UniqueName: \"kubernetes.io/projected/b10257bd-92be-4617-beab-f2270c5d396d-kube-api-access-bc7xd\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670153 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d5c0784-95be-40af-98df-14e58bf411f0-crio-socket\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670176 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670241 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-env-overrides\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670246 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670285 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670316 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc79x\" (UniqueName: \"kubernetes.io/projected/889ada6d-d4ee-4f5e-aa22-23637608940a-kube-api-access-zc79x\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670343 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670373 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-var-lib-kubelet\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670399 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-cnibin\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670433 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cad3e-0d13-4459-b970-79675d07443d-ovn-node-metrics-cert\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670461 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-tls\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.670565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670485 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-modprobe-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670602 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670666 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670679 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cad3e-0d13-4459-b970-79675d07443d-ovn-node-metrics-cert\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670693 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670698 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670719 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-cni-binary-copy\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670743 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-var-lib-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670769 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7tj\" (UniqueName: \"kubernetes.io/projected/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kube-api-access-cb7tj\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670813 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9tj\" (UniqueName: \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670839 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b136c4-f8d9-4b45-9454-ab09db908f72-serviceca\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670863 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670888 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d5c0784-95be-40af-98df-14e58bf411f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670913 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-multus\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670917 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4f8473-1c87-463e-8338-ee1971bd0064-cni-binary-copy\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670937 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-hostroot\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.670980 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77j56\" (UniqueName: \"kubernetes.io/projected/4f4cad3e-0d13-4459-b970-79675d07443d-kube-api-access-77j56\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671036 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhsz\" (UniqueName: \"kubernetes.io/projected/a49fe757-54a2-4871-b450-d48e15571c9c-kube-api-access-jrhsz\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671054 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.671066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671074 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-run\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671090 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671121 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-kubelet\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671144 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/befe88bf-8d8b-47b8-a244-06524e4c3e28-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671166 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxbjn\" (UniqueName: \"kubernetes.io/projected/7d5c0784-95be-40af-98df-14e58bf411f0-kube-api-access-vxbjn\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671210 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671231 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/befe88bf-8d8b-47b8-a244-06524e4c3e28-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671249 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671286 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671294 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671321 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-registration-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671374 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysconfig\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671424 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-etc-kubernetes\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671440 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-netd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671457 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671490 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-socket-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671591 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671657 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-system-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671688 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671713 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671752 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-config\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671773 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-root\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671802 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b136c4-f8d9-4b45-9454-ab09db908f72-serviceca\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671807 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10df746e-9937-45a2-ae81-c39ddc189ea6-tmp-dir\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671848 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2j52\" (UniqueName: \"kubernetes.io/projected/10df746e-9937-45a2-ae81-c39ddc189ea6-kube-api-access-m2j52\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:15:33.671859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671860 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10df746e-9937-45a2-ae81-c39ddc189ea6-tmp-dir\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671877 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1e57895-4080-4a05-a5b0-b859cdfd96dc-host-slash\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671904 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7736e260-9632-4568-a43c-33b5eecd6acb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671905 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smh4k\" (UniqueName: \"kubernetes.io/projected/59fd2087-a50b-4e0a-8439-379d3094ee59-kube-api-access-smh4k\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671947 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d5c0784-95be-40af-98df-14e58bf411f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671973 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671974 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cad3e-0d13-4459-b970-79675d07443d-ovnkube-config\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.671999 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672025 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9kr\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672070 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-lib-modules\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672093 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b136c4-f8d9-4b45-9454-ab09db908f72-host\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672144 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02c1435f-0c51-4647-b1c2-0334a39a1af2-agent-certs\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672179 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1e57895-4080-4a05-a5b0-b859cdfd96dc-iptables-alerter-script\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672196 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/889ada6d-d4ee-4f5e-aa22-23637608940a-cert\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672220 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-device-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672346 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1e57895-4080-4a05-a5b0-b859cdfd96dc-iptables-alerter-script\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:15:33.672639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.672425 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02c1435f-0c51-4647-b1c2-0334a39a1af2-agent-certs\") pod \"konnectivity-agent-6ljk7\" (UID: \"02c1435f-0c51-4647-b1c2-0334a39a1af2\") " pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:15:33.710318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.710268 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-46jbj\"" Nov 03 21:15:33.710937 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.710919 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4hvm9\"" Nov 03 21:15:33.710937 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.710919 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:15:33.711046 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.710965 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 21:15:33.711172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711159 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-82cgq4d3kfu1j\"" Nov 03 21:15:33.711172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711165 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 21:15:33.711242 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711174 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 21:15:33.711386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711366 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-95lxt\"" Nov 03 21:15:33.711483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711420 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 21:15:33.711483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711429 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rf5\"" Nov 03 21:15:33.711483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711455 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 21:15:33.711483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711480 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 21:15:33.711700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711594 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 21:15:33.711700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711654 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.711700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711664 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 21:15:33.712030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711938 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.712030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711952 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 21:15:33.712030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711958 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qqtpl\"" Nov 03 21:15:33.712030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711983 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 21:15:33.712030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711994 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 21:15:33.712030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.711979 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 21:15:33.712261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712118 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79p29\"" Nov 03 21:15:33.712261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712151 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 21:15:33.712261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712162 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 21:15:33.712261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712215 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 21:15:33.712261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712232 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kz2rr\"" Nov 03 21:15:33.712443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712376 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/889ada6d-d4ee-4f5e-aa22-23637608940a-cert\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:33.712443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712399 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 21:15:33.712514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712455 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 21:15:33.712649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712635 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 21:15:33.712708 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712691 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.713040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712937 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 21:15:33.713040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712953 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 21:15:33.713040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712968 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 21:15:33.713040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712988 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 21:15:33.713040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.712989 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:15:33.713040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713021 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 21:15:33.713353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713054 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 21:15:33.713353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713193 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 21:15:33.713353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713209 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 21:15:33.713353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713260 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 21:15:33.713353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713262 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 21:15:33.713353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713264 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 21:15:33.713353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713263 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713358 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713411 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713447 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713504 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713531 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8l7\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713528 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713542 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713534 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713543 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713549 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-14esuk0guf2o7\"" Nov 03 21:15:33.713569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713574 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713581 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713700 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713720 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713736 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713748 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713750 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713768 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713786 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713789 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cue34h59ah6oo\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713811 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713816 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ftm6x\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713824 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 21:15:33.713981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.713864 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 21:15:33.717196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.717181 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.717282 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.717271 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.717395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.717381 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.717748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.717733 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.717902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.717888 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.717947 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.717908 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.719053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719038 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b10257bd-92be-4617-beab-f2270c5d396d-metrics-tls\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.719110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719040 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.719196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719183 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.719259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719211 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.719311 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719298 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.719363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719335 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b10257bd-92be-4617-beab-f2270c5d396d-config-volume\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.719415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719408 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.719553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719538 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.719626 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.719566 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.720301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.720286 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.720346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.720321 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-grpc-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.720393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.720350 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-metrics-client-ca\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.720393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.720360 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d5c977-b290-45f5-a219-8f2a98eab0f6-metrics-client-ca\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.720393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.720378 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.720485 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.720460 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.720587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.720572 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.721523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.721506 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d5c0784-95be-40af-98df-14e58bf411f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.721594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.721520 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.721594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.721569 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.721594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.721586 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.721767 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.721641 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.721814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.721774 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.722759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.722746 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.722759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.722751 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.722759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.722753 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58d5c977-b290-45f5-a219-8f2a98eab0f6-secret-thanos-querier-tls\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.759741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.759716 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 21:15:33.759840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.759739 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 21:15:33.759885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.759837 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tq7pf\"" Nov 03 21:15:33.760701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.760685 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4ccvj\"" Nov 03 21:15:33.760752 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.760699 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 21:15:33.760752 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.760731 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-tls\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.761001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.760985 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 21:15:33.761851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.761832 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.762909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.762299 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 21:15:33.762909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.762408 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d5c0784-95be-40af-98df-14e58bf411f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.766147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.763892 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 21:15:33.766147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.764294 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 21:15:33.766147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.764922 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pfj\" (UniqueName: \"kubernetes.io/projected/01b136c4-f8d9-4b45-9454-ab09db908f72-kube-api-access-67pfj\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:15:33.766147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.765239 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 21:15:33.766147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.765667 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmvs\" (UniqueName: \"kubernetes.io/projected/58d5c977-b290-45f5-a219-8f2a98eab0f6-kube-api-access-kfmvs\") pod \"thanos-querier-77f7cb99c4-tz5gh\" (UID: \"58d5c977-b290-45f5-a219-8f2a98eab0f6\") " pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:33.766147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.765808 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77j56\" (UniqueName: \"kubernetes.io/projected/4f4cad3e-0d13-4459-b970-79675d07443d-kube-api-access-77j56\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.766888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.766663 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fld\" (UniqueName: \"kubernetes.io/projected/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-api-access-p9fld\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.766888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.766699 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wln6\" (UniqueName: \"kubernetes.io/projected/e1e57895-4080-4a05-a5b0-b859cdfd96dc-kube-api-access-4wln6\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:15:33.766888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.766806 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6f4\" (UniqueName: \"kubernetes.io/projected/7736e260-9632-4568-a43c-33b5eecd6acb-kube-api-access-rc6f4\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.767096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767080 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7xd\" (UniqueName: \"kubernetes.io/projected/b10257bd-92be-4617-beab-f2270c5d396d-kube-api-access-bc7xd\") pod \"dns-default-9srkw\" (UID: \"b10257bd-92be-4617-beab-f2270c5d396d\") " pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:33.767157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767106 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxbjn\" (UniqueName: \"kubernetes.io/projected/7d5c0784-95be-40af-98df-14e58bf411f0-kube-api-access-vxbjn\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.767157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767117 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncr4\" (UniqueName: \"kubernetes.io/projected/5a4f8473-1c87-463e-8338-ee1971bd0064-kube-api-access-5ncr4\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.767157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767143 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7tj\" (UniqueName: \"kubernetes.io/projected/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kube-api-access-cb7tj\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.767301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767256 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpr2d\" (UniqueName: \"kubernetes.io/projected/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-kube-api-access-vpr2d\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.767498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767476 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrsz\" (UniqueName: \"kubernetes.io/projected/5e7459ce-aa09-4c74-b5ef-8424cb8d24b5-kube-api-access-kvrsz\") pod \"openshift-state-metrics-7bd7489957-62h4t\" (UID: \"5e7459ce-aa09-4c74-b5ef-8424cb8d24b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-62h4t" Nov 03 21:15:33.767561 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767502 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhsz\" (UniqueName: \"kubernetes.io/projected/a49fe757-54a2-4871-b450-d48e15571c9c-kube-api-access-jrhsz\") pod \"network-metrics-daemon-nkqjv\" (UID: \"a49fe757-54a2-4871-b450-d48e15571c9c\") " pod="openshift-multus/network-metrics-daemon-nkqjv" Nov 03 21:15:33.767664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767644 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smh4k\" (UniqueName: \"kubernetes.io/projected/59fd2087-a50b-4e0a-8439-379d3094ee59-kube-api-access-smh4k\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.767768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767744 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2j52\" (UniqueName: \"kubernetes.io/projected/10df746e-9937-45a2-ae81-c39ddc189ea6-kube-api-access-m2j52\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:15:33.767839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.767788 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fhz\" (UniqueName: \"kubernetes.io/projected/39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de-kube-api-access-w4fhz\") pod \"network-check-target-2xkcf\" (UID: \"39f4aeef-6f5d-48bc-8bbe-bb7e8e5658de\") " pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:33.768023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.768001 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/befe88bf-8d8b-47b8-a244-06524e4c3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-458sb\" (UID: \"befe88bf-8d8b-47b8-a244-06524e4c3e28\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-458sb" Nov 03 21:15:33.768490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.768468 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.768549 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.768476 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc79x\" (UniqueName: \"kubernetes.io/projected/889ada6d-d4ee-4f5e-aa22-23637608940a-kube-api-access-zc79x\") pod \"ingress-canary-thpr2\" (UID: \"889ada6d-d4ee-4f5e-aa22-23637608940a\") " pod="openshift-ingress-canary/ingress-canary-thpr2" Nov 03 21:15:33.768653 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.768634 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9tj\" (UniqueName: \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\") pod \"console-789d48d68-988j4\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:33.768713 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.768640 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9kr\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\") pod \"alertmanager-main-0\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:33.773465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773447 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.773503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773473 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-k8s-cni-cncf-io\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.773503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773489 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-conf-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.773598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773502 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.773598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773516 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.773598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773539 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-k8s-cni-cncf-io\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.773598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773556 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.773598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773558 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-conf-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.773598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773587 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773650 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-systemd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773671 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773674 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10df746e-9937-45a2-ae81-c39ddc189ea6-hosts-file\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773677 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773706 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmw5\" (UniqueName: \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773725 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-systemd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773740 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10df746e-9937-45a2-ae81-c39ddc189ea6-hosts-file\") pod \"node-resolver-rvzp6\" (UID: \"10df746e-9937-45a2-ae81-c39ddc189ea6\") " pod="openshift-dns/node-resolver-rvzp6" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773744 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773775 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773820 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-host\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.773858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773835 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773867 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-kubelet-config\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773898 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-kubelet-config\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773904 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-host\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773924 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773941 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773961 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-cnibin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.773983 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-sys\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774005 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774027 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774049 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-kubernetes\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774064 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-sys\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774071 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-kubelet\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774092 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774098 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-cnibin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774110 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-kubernetes\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774115 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774140 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-kubelet\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774179 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774180 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-bin\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774216 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-bin\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774220 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-socket-dir-parent\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774224 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774242 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774251 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774260 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.774276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774254 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-tls\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774291 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-socket-dir-parent\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774306 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-slash\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774332 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-ovn\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774341 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774357 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774376 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-conf\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774393 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d5c0784-95be-40af-98df-14e58bf411f0-crio-socket\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774394 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-slash\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774407 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774413 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-run-ovn\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774422 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-var-lib-kubelet\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774436 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-tls\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774436 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-cnibin\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774457 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d5c0784-95be-40af-98df-14e58bf411f0-crio-socket\") pod \"insights-runtime-extractor-xcf7l\" (UID: \"7d5c0784-95be-40af-98df-14e58bf411f0\") " pod="openshift-insights/insights-runtime-extractor-xcf7l" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774471 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-cnibin\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774471 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-modprobe-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774496 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-multus-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774507 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-var-lib-kubelet\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774503 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-var-lib-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774534 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774543 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-var-lib-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774551 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-modprobe-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774547 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-client-certs\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774563 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-conf\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774581 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774597 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774665 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-multus\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774688 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-hostroot\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774707 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774720 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f070dd5b-30ed-4531-8cf6-196779398aad-audit-log\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774736 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-run\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774776 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-hostroot\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774782 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-run\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774780 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-secret-metrics-server-client-certs\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774800 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774809 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774803 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774824 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f070dd5b-30ed-4531-8cf6-196779398aad-audit-log\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774828 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-multus\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774804 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-kubelet\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774827 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-kubelet\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774870 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-registration-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774894 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysconfig\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774917 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-registration-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774917 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-etc-kubernetes\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774945 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-netd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774949 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysconfig\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774947 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-etc-kubernetes\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774960 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-socket-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774975 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774982 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-cni-netd\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.774993 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-system-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775017 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-root\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775034 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-socket-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775054 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775057 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-system-cni-dir\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775071 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775085 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1e57895-4080-4a05-a5b0-b859cdfd96dc-host-slash\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775102 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775115 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-lib-modules\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775128 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b136c4-f8d9-4b45-9454-ab09db908f72-host\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775137 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1e57895-4080-4a05-a5b0-b859cdfd96dc-host-slash\") pod \"iptables-alerter-jqr6h\" (UID: \"e1e57895-4080-4a05-a5b0-b859cdfd96dc\") " pod="openshift-network-operator/iptables-alerter-jqr6h" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775056 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-root\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775152 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775165 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-device-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775181 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775188 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775197 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b136c4-f8d9-4b45-9454-ab09db908f72-host\") pod \"node-ca-lrwcd\" (UID: \"01b136c4-f8d9-4b45-9454-ab09db908f72\") " pod="openshift-image-registry/node-ca-lrwcd" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775210 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-sys\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775227 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-lib-modules\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775234 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-etc-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775212 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-device-dir\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775251 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775259 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-node-log\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775276 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-etc-openvswitch\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775284 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775303 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775310 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-node-log\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775284 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-sys\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775348 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775366 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-netns\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775379 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-systemd\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775378 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775388 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-sysctl-d\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775394 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-log-socket\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775405 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775421 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-log-socket\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775455 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59fd2087-a50b-4e0a-8439-379d3094ee59-etc-systemd\") pod \"tuned-lwqh9\" (UID: \"59fd2087-a50b-4e0a-8439-379d3094ee59\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqh9" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775421 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-run-netns\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.775400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775420 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-client-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775499 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775525 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775549 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvvz\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775575 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6rw\" (UniqueName: \"kubernetes.io/projected/f070dd5b-30ed-4531-8cf6-196779398aad-kube-api-access-tz6rw\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775580 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f070dd5b-30ed-4531-8cf6-196779398aad-client-ca-bundle\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775599 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-multus-certs\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775653 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775660 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775686 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775713 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-bin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775689 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-multus-certs\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775740 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775794 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775797 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-var-lib-cni-bin\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775817 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775839 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-netns\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775866 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-systemd-units\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775889 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775838 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775914 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775911 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-metrics-server-audit-profiles\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775952 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775964 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cad3e-0d13-4459-b970-79675d07443d-systemd-units\") pod \"ovnkube-node-x8bjk\" (UID: \"4f4cad3e-0d13-4459-b970-79675d07443d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775979 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.775955 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4f8473-1c87-463e-8338-ee1971bd0064-host-run-netns\") pod \"multus-5wwt4\" (UID: \"5a4f8473-1c87-463e-8338-ee1971bd0064\") " pod="openshift-multus/multus-5wwt4" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776001 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776004 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-system-cni-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776037 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-wtmp\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776046 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776037 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7736e260-9632-4568-a43c-33b5eecd6acb-system-cni-dir\") pod \"multus-additional-cni-plugins-qjndk\" (UID: \"7736e260-9632-4568-a43c-33b5eecd6acb\") " pod="openshift-multus/multus-additional-cni-plugins-qjndk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776062 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-sys-fs\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776086 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fffd8c3e-675e-45b0-b22f-445e5a92bbfd-node-exporter-wtmp\") pod \"node-exporter-tbv9l\" (UID: \"fffd8c3e-675e-45b0-b22f-445e5a92bbfd\") " pod="openshift-monitoring/node-exporter-tbv9l" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776094 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776104 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e346f0cc-7ed6-42f5-a4f9-1c16320cefd9-sys-fs\") pod \"aws-ebs-csi-driver-node-gd2rb\" (UID: \"e346f0cc-7ed6-42f5-a4f9-1c16320cefd9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd2rb" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776105 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776119 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776166 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776200 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f070dd5b-30ed-4531-8cf6-196779398aad-metrics-server-audit-profiles\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776264 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776270 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-dbus\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776297 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776340 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8acde6b-bb34-4c16-9afe-aa4c21f59dd6-dbus\") pod \"global-pull-secret-syncer-9vfbk\" (UID: \"d8acde6b-bb34-4c16-9afe-aa4c21f59dd6\") " pod="kube-system/global-pull-secret-syncer-9vfbk" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776377 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776392 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.778144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.776860 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:33.829220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.829194 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmw5\" (UniqueName: \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\") pod \"console-9797677c8-8gnvt\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:33.830086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.830064 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6rw\" (UniqueName: \"kubernetes.io/projected/f070dd5b-30ed-4531-8cf6-196779398aad-kube-api-access-tz6rw\") pod \"metrics-server-ff498dd48-lmxq5\" (UID: \"f070dd5b-30ed-4531-8cf6-196779398aad\") " pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:33.830171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:33.830118 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvvz\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\") pod \"prometheus-k8s-0\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:34.222350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:34.222310 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:34.457730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:34.457696 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:34.504308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:34.504249 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9srkw" Nov 03 21:15:35.028004 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:35.027977 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:35.043482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:35.043457 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:35.129449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:35.129413 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:35.261161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:35.261132 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-789d48d68-988j4"] Nov 03 21:15:35.345909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:35.345873 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:35.596055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:35.595947 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:36.749432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:36.749404 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:36.933542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:36.933513 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ljk7" Nov 03 21:15:37.772751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:37.772710 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:37.776204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:37.776185 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:38.228293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:38.228263 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77f7cb99c4-tz5gh" Nov 03 21:15:40.049392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:40.049360 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:40.049867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:40.049604 5931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:15:40.690624 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:40.690588 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:15:41.067579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:41.067447 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:41.295652 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:41.295583 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:41.642448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:41.642405 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:15:42.768281 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:42.768250 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:42.992555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:42.992521 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:43.017050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:43.017022 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:44.071533 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:44.071501 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:44.299057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:44.299025 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:45.541349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:45.541315 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:46.735039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:46.734998 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:46.955475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:46.955420 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:48.018066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:48.018028 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8bjk" Nov 03 21:15:49.134850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:49.134810 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:49.360638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:49.360580 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:51.175558 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.175505 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:51.401473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.401433 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:51.646144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.646109 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:15:51.646437 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.646410 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="alertmanager" containerID="cri-o://11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d" gracePeriod=120 Nov 03 21:15:51.646532 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.646447 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-metric" containerID="cri-o://6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00" gracePeriod=120 Nov 03 21:15:51.646532 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.646463 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="prom-label-proxy" containerID="cri-o://aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42" gracePeriod=120 Nov 03 21:15:51.646685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.646514 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-web" containerID="cri-o://2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb" gracePeriod=120 Nov 03 21:15:51.646685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.646515 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy" containerID="cri-o://1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c" gracePeriod=120 Nov 03 21:15:51.646786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:51.646566 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="config-reloader" containerID="cri-o://e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a" gracePeriod=120 Nov 03 21:15:52.387029 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.386999 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-ff498dd48-lmxq5" Nov 03 21:15:52.648875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648780 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42" exitCode=0 Nov 03 21:15:52.648875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648811 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00" exitCode=0 Nov 03 21:15:52.648875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648819 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c" exitCode=0 Nov 03 21:15:52.648875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648817 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42"} Nov 03 21:15:52.649179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648897 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00"} Nov 03 21:15:52.649179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648935 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c"} Nov 03 21:15:52.649179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648971 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a"} Nov 03 21:15:52.649179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.648829 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a" exitCode=0 Nov 03 21:15:52.649179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.649016 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d" exitCode=0 Nov 03 21:15:52.649179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.649079 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d"} Nov 03 21:15:52.928867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:52.928798 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:53.022012 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.021985 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:15:53.049347 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049307 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049371 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049400 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049434 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049452 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049471 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049815 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049496 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049872 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049822 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:53.049921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049879 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.049981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049927 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.050190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.049980 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.050190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.050018 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.050190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.050079 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.050342 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.050177 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\") pod \"f9816356-82fa-42f8-b5b4-b5fd84a32042\" (UID: \"f9816356-82fa-42f8-b5b4-b5fd84a32042\") " Nov 03 21:15:53.050426 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.050398 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:53.050842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.050537 5931 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-main-db\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.050842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.050564 5931 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.051123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.051094 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:53.052784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.052741 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:53.053385 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.053361 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:53.053536 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.053506 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:53.053697 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.053676 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "kube-api-access-dw9kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:53.053772 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.053695 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:53.053960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.053939 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:53.054175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.054149 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:53.054260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.054210 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:53.057664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.057642 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:53.064629 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.064594 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config" pod "f9816356-82fa-42f8-b5b4-b5fd84a32042" (UID: "f9816356-82fa-42f8-b5b4-b5fd84a32042"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:53.151843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151811 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9kr\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-kube-api-access-dw9kr\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.151843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151847 5931 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-out\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151864 5931 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-cluster-tls-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151879 5931 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9816356-82fa-42f8-b5b4-b5fd84a32042-tls-assets\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151901 5931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151918 5931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151934 5931 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-web-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151947 5931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151960 5931 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151974 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9816356-82fa-42f8-b5b4-b5fd84a32042-metrics-client-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.152080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.151987 5931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9816356-82fa-42f8-b5b4-b5fd84a32042-secret-alertmanager-main-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:53.655577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.655538 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerID="2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb" exitCode=0 Nov 03 21:15:53.656080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.655590 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb"} Nov 03 21:15:53.656080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.655663 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:15:53.656080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.655686 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9816356-82fa-42f8-b5b4-b5fd84a32042","Type":"ContainerDied","Data":"47178319b642438d2e7caad403e1f39f1b444016d712a8c1d323d5ae68375d2c"} Nov 03 21:15:53.656080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.655717 5931 scope.go:122] "RemoveContainer" containerID="aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42" Nov 03 21:15:53.663893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.663874 5931 scope.go:122] "RemoveContainer" containerID="6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00" Nov 03 21:15:53.670676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.670661 5931 scope.go:122] "RemoveContainer" containerID="1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c" Nov 03 21:15:53.677247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.677227 5931 scope.go:122] "RemoveContainer" containerID="2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb" Nov 03 21:15:53.685747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.685729 5931 scope.go:122] "RemoveContainer" containerID="e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a" Nov 03 21:15:53.692531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.692512 5931 scope.go:122] "RemoveContainer" containerID="11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d" Nov 03 21:15:53.699224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.699208 5931 scope.go:122] "RemoveContainer" containerID="9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd" Nov 03 21:15:53.705812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.705796 5931 scope.go:122] "RemoveContainer" containerID="aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42" Nov 03 21:15:53.706080 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:53.706059 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42\": container with ID starting with aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42 not found: ID does not exist" containerID="aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42" Nov 03 21:15:53.706125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706091 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42"} err="failed to get container status \"aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42\": rpc error: code = NotFound desc = could not find container \"aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42\": container with ID starting with aa3c81591cdd90656ab00dd71d3331b1e2430470c21b6a8f3133fb69fe6aab42 not found: ID does not exist" Nov 03 21:15:53.706170 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706127 5931 scope.go:122] "RemoveContainer" containerID="6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00" Nov 03 21:15:53.706348 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:53.706331 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00\": container with ID starting with 6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00 not found: ID does not exist" containerID="6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00" Nov 03 21:15:53.706389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706355 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00"} err="failed to get container status \"6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00\": rpc error: code = NotFound desc = could not find container \"6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00\": container with ID starting with 6276076cde8f9a9138469d99bedd196e42a514adf5f6438532e03ef0a0b6dd00 not found: ID does not exist" Nov 03 21:15:53.706389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706369 5931 scope.go:122] "RemoveContainer" containerID="1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c" Nov 03 21:15:53.706571 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:53.706556 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c\": container with ID starting with 1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c not found: ID does not exist" containerID="1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c" Nov 03 21:15:53.706603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706577 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c"} err="failed to get container status \"1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c\": rpc error: code = NotFound desc = could not find container \"1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c\": container with ID starting with 1be6d96efd3d472a0620edfff384aa3acd090f57256057d20ad4275ba9a62c5c not found: ID does not exist" Nov 03 21:15:53.706603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706590 5931 scope.go:122] "RemoveContainer" containerID="2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb" Nov 03 21:15:53.706822 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:53.706805 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb\": container with ID starting with 2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb not found: ID does not exist" containerID="2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb" Nov 03 21:15:53.706876 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706825 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb"} err="failed to get container status \"2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb\": rpc error: code = NotFound desc = could not find container \"2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb\": container with ID starting with 2c08ec0f8e5c8cd853787e4df1809706e3398d9a93d3facbfeeee729f26fe2fb not found: ID does not exist" Nov 03 21:15:53.706876 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.706837 5931 scope.go:122] "RemoveContainer" containerID="e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a" Nov 03 21:15:53.707045 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:53.707029 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a\": container with ID starting with e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a not found: ID does not exist" containerID="e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a" Nov 03 21:15:53.707100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.707049 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a"} err="failed to get container status \"e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a\": rpc error: code = NotFound desc = could not find container \"e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a\": container with ID starting with e18b8497bc04dd720db6e2bd647f60a8c1f0e78b18edeba3545aa5242a55889a not found: ID does not exist" Nov 03 21:15:53.707100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.707061 5931 scope.go:122] "RemoveContainer" containerID="11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d" Nov 03 21:15:53.707266 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:53.707247 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d\": container with ID starting with 11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d not found: ID does not exist" containerID="11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d" Nov 03 21:15:53.707326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.707274 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d"} err="failed to get container status \"11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d\": rpc error: code = NotFound desc = could not find container \"11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d\": container with ID starting with 11cdbebaa593017ebd7e8d796ed7fd6fcc35441898f774f4c6e0b72fc2228c0d not found: ID does not exist" Nov 03 21:15:53.707326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.707293 5931 scope.go:122] "RemoveContainer" containerID="9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd" Nov 03 21:15:53.707535 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:15:53.707520 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd\": container with ID starting with 9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd not found: ID does not exist" containerID="9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd" Nov 03 21:15:53.707578 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.707538 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd"} err="failed to get container status \"9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd\": rpc error: code = NotFound desc = could not find container \"9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd\": container with ID starting with 9c9a2bbb0fd7bb8460b1a8170611b3bfb5f6349b0eba3efa20a75f0201e31bbd not found: ID does not exist" Nov 03 21:15:53.821705 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.821678 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:15:53.871540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:53.871439 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:15:54.503731 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:54.503700 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" path="/var/lib/kubelet/pods/f9816356-82fa-42f8-b5b4-b5fd84a32042/volumes" Nov 03 21:15:55.263921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:55.263880 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:55.496875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:55.496845 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:56.077989 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:56.077949 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:56.312042 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:56.312007 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:56.549436 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:56.549404 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:56.645101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:56.645068 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9797677c8-8gnvt"] Nov 03 21:15:57.041819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.041781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:57.267566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.267536 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:57.447847 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.447815 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:57.448292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.448112 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="prometheus" containerID="cri-o://b65817f7be82b8a646f4808732af1ae04b9e6a1921fced7d52f1a6168a41721a" gracePeriod=600 Nov 03 21:15:57.448292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.448164 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy" containerID="cri-o://18a30cd7dc300c546d6acad101776849033be8d61c2af4e549b402de8e2626b9" gracePeriod=600 Nov 03 21:15:57.448292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.448188 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-web" containerID="cri-o://4f9dc7f4c4b16bf502a27e1482295444e6ca2e6b1cea68b03873d385a7ea3c52" gracePeriod=600 Nov 03 21:15:57.448292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.448161 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="thanos-sidecar" containerID="cri-o://85bdc4954edc07d2accef1f8e7d806c92f99bad57a713dea15fd3b973511da87" gracePeriod=600 Nov 03 21:15:57.448292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.448278 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="config-reloader" containerID="cri-o://48a0135fd7b35e2f309129d181404c5b11b07beea0316520a87de39467bc245c" gracePeriod=600 Nov 03 21:15:57.448543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.448341 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c8ac7fad04959805aef82621347439cfe06af1b68d2925a8187835b7ce95c333" gracePeriod=600 Nov 03 21:15:57.603519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.603485 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:57.680553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680521 5931 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="c8ac7fad04959805aef82621347439cfe06af1b68d2925a8187835b7ce95c333" exitCode=0 Nov 03 21:15:57.680553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680554 5931 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="18a30cd7dc300c546d6acad101776849033be8d61c2af4e549b402de8e2626b9" exitCode=0 Nov 03 21:15:57.680729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680568 5931 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="4f9dc7f4c4b16bf502a27e1482295444e6ca2e6b1cea68b03873d385a7ea3c52" exitCode=0 Nov 03 21:15:57.680729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680580 5931 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="85bdc4954edc07d2accef1f8e7d806c92f99bad57a713dea15fd3b973511da87" exitCode=0 Nov 03 21:15:57.680729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680589 5931 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="48a0135fd7b35e2f309129d181404c5b11b07beea0316520a87de39467bc245c" exitCode=0 Nov 03 21:15:57.680729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680587 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"c8ac7fad04959805aef82621347439cfe06af1b68d2925a8187835b7ce95c333"} Nov 03 21:15:57.680729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680599 5931 generic.go:356] "Generic (PLEG): container finished" podID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerID="b65817f7be82b8a646f4808732af1ae04b9e6a1921fced7d52f1a6168a41721a" exitCode=0 Nov 03 21:15:57.680729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680703 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"18a30cd7dc300c546d6acad101776849033be8d61c2af4e549b402de8e2626b9"} Nov 03 21:15:57.680918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680781 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"4f9dc7f4c4b16bf502a27e1482295444e6ca2e6b1cea68b03873d385a7ea3c52"} Nov 03 21:15:57.680918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680826 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"85bdc4954edc07d2accef1f8e7d806c92f99bad57a713dea15fd3b973511da87"} Nov 03 21:15:57.680918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680863 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"48a0135fd7b35e2f309129d181404c5b11b07beea0316520a87de39467bc245c"} Nov 03 21:15:57.680918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.680900 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"b65817f7be82b8a646f4808732af1ae04b9e6a1921fced7d52f1a6168a41721a"} Nov 03 21:15:57.691356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.691338 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:57.792101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792033 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792072 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792090 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792111 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792139 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792164 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792194 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792238 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792267 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792495 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792503 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:57.792666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792549 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792562 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:57.792666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792577 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792637 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792670 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792712 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792747 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.792954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792829 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.793142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.792980 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.793142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.793010 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.793142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.793056 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\") pod \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\" (UID: \"bed2f3a9-343c-4a13-a9d8-68dafc4472fb\") " Nov 03 21:15:57.793334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.793313 5931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.793403 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.793342 5931 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.794210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.794095 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:57.794766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.794353 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:57.794766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.794542 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:57.794766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.794725 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:57.795131 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.795107 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "kube-api-access-sdvvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:57.795217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.795176 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:57.795217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.795194 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:57.795473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.795451 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:57.796667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.796641 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:57.796803 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.796784 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.797068 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.797035 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.797362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.797340 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.797465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.797428 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.797559 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.797541 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.797977 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.797956 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.798100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.798077 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.798211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.798195 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.806741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.806722 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config" pod "bed2f3a9-343c-4a13-a9d8-68dafc4472fb" (UID: "bed2f3a9-343c-4a13-a9d8-68dafc4472fb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:57.894544 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894515 5931 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894544 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894542 5931 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894552 5931 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894563 5931 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894572 5931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-db\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894580 5931 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894589 5931 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-configmap-metrics-client-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894600 5931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894620 5931 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894629 5931 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894637 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvvz\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-kube-api-access-sdvvz\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894645 5931 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-tls-assets\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894653 5931 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-kube-rbac-proxy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894661 5931 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-config-out\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894669 5931 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-metrics-client-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894677 5931 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-web-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894684 5931 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-secret-grpc-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.894745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:57.894692 5931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bed2f3a9-343c-4a13-a9d8-68dafc4472fb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:58.152791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.152749 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:58.687264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.687234 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bed2f3a9-343c-4a13-a9d8-68dafc4472fb","Type":"ContainerDied","Data":"adac3c9dac7b0bb2fbe7fa032024c277460ab64ee37870b70d284cbb58553e73"} Nov 03 21:15:58.687723 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.687278 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:58.687723 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.687307 5931 scope.go:122] "RemoveContainer" containerID="c8ac7fad04959805aef82621347439cfe06af1b68d2925a8187835b7ce95c333" Nov 03 21:15:58.694826 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.694806 5931 scope.go:122] "RemoveContainer" containerID="18a30cd7dc300c546d6acad101776849033be8d61c2af4e549b402de8e2626b9" Nov 03 21:15:58.701699 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.701682 5931 scope.go:122] "RemoveContainer" containerID="4f9dc7f4c4b16bf502a27e1482295444e6ca2e6b1cea68b03873d385a7ea3c52" Nov 03 21:15:58.708074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.708053 5931 scope.go:122] "RemoveContainer" containerID="85bdc4954edc07d2accef1f8e7d806c92f99bad57a713dea15fd3b973511da87" Nov 03 21:15:58.714303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.714289 5931 scope.go:122] "RemoveContainer" containerID="48a0135fd7b35e2f309129d181404c5b11b07beea0316520a87de39467bc245c" Nov 03 21:15:58.720889 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.720870 5931 scope.go:122] "RemoveContainer" containerID="b65817f7be82b8a646f4808732af1ae04b9e6a1921fced7d52f1a6168a41721a" Nov 03 21:15:58.727551 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.727532 5931 scope.go:122] "RemoveContainer" containerID="1e0348bb86e8b11e81b7d2aab96b40e40ad38da22be1d0969cd611165dee501e" Nov 03 21:15:58.901745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.901715 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:58.951584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.951548 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:58.952086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952057 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:58.952457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952437 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="alertmanager" Nov 03 21:15:58.952518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952472 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="alertmanager" Nov 03 21:15:58.952518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952488 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-web" Nov 03 21:15:58.952518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952496 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-web" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952518 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952525 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952545 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="thanos-sidecar" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952553 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="thanos-sidecar" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952564 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="config-reloader" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952568 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="config-reloader" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952574 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="prometheus" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952577 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="prometheus" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952584 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="init-config-reloader" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952589 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="init-config-reloader" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952595 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952599 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952635 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="init-config-reloader" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952642 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="init-config-reloader" Nov 03 21:15:58.952646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952651 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="prom-label-proxy" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952656 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="prom-label-proxy" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952663 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-web" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952667 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-web" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952673 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-metric" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952677 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-metric" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952684 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-thanos" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952688 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-thanos" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952693 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="config-reloader" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952697 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="config-reloader" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952742 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952750 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="thanos-sidecar" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952756 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="config-reloader" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952761 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="prometheus" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952766 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="init-config-reloader" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952771 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952776 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="init-config-reloader" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952782 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="prom-label-proxy" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952787 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-web" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952791 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-metric" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952796 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" containerName="kube-rbac-proxy-thanos" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952802 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="config-reloader" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952807 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="alertmanager" Nov 03 21:15:58.953129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.952812 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9816356-82fa-42f8-b5b4-b5fd84a32042" containerName="kube-rbac-proxy-web" Nov 03 21:15:58.956016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:58.956001 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.008446 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.008376 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 21:15:59.008792 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.008773 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 21:15:59.009346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.009315 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 21:15:59.009429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.009366 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 21:15:59.009494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.009471 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 21:15:59.009494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.009487 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 21:15:59.009629 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.009596 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-14esuk0guf2o7\"" Nov 03 21:15:59.058174 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.058151 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 21:15:59.058174 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.058151 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 21:15:59.058333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.058265 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 21:15:59.058378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.058340 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 21:15:59.058521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.058507 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ftm6x\"" Nov 03 21:15:59.058557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.058508 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 21:15:59.059218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.059200 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 21:15:59.059307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.059295 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 21:15:59.102654 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102628 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102654 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102658 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102679 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102770 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102796 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102815 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102869 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbcj\" (UniqueName: \"kubernetes.io/projected/cc89e350-e328-41e4-afd4-c55a6d4ae69a-kube-api-access-pqbcj\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102896 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.102952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102920 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.102969 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103020 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103036 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103053 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103069 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-config\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103091 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc89e350-e328-41e4-afd4-c55a6d4ae69a-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103184 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103147 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103184 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103178 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103242 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103207 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc89e350-e328-41e4-afd4-c55a6d4ae69a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103247 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.103299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.103268 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.108438 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.108424 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 21:15:59.182809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.182757 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:59.204154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204130 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204162 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204187 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204201 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204218 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204234 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-config\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204254 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc89e350-e328-41e4-afd4-c55a6d4ae69a-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204399 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204431 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204461 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc89e350-e328-41e4-afd4-c55a6d4ae69a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204478 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204504 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204529 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204552 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204558 5931 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 21:15:59.204581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204574 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.205045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204634 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.205045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204664 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.205045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204691 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.205045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204743 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbcj\" (UniqueName: \"kubernetes.io/projected/cc89e350-e328-41e4-afd4-c55a6d4ae69a-kube-api-access-pqbcj\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.205045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204772 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.205045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204889 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.205045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.204929 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.206834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.206225 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.206834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.206388 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.207275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.207224 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.208002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.207939 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.208117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.208093 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.208313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.208289 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc89e350-e328-41e4-afd4-c55a6d4ae69a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.208887 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.208861 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.208994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.208876 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.209091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.209005 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc89e350-e328-41e4-afd4-c55a6d4ae69a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.209196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.209100 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc89e350-e328-41e4-afd4-c55a6d4ae69a-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.209513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.209481 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-config\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.210304 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.210286 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.210449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.210427 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.210554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.210536 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.210725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.210708 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.211060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.211035 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.211232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.211209 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc89e350-e328-41e4-afd4-c55a6d4ae69a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.262791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.262734 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbcj\" (UniqueName: \"kubernetes.io/projected/cc89e350-e328-41e4-afd4-c55a6d4ae69a-kube-api-access-pqbcj\") pod \"prometheus-k8s-0\" (UID: \"cc89e350-e328-41e4-afd4-c55a6d4ae69a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.271559 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.271535 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:15:59.696308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:15:59.696267 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 21:15:59.700856 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:15:59.700817 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc89e350_e328_41e4_afd4_c55a6d4ae69a.slice/crio-2c038394debb5cc166ec79fdeb4084d4991cdc47656f13549acf42461f3bcd65 WatchSource:0}: Error finding container 2c038394debb5cc166ec79fdeb4084d4991cdc47656f13549acf42461f3bcd65: Status 404 returned error can't find the container with id 2c038394debb5cc166ec79fdeb4084d4991cdc47656f13549acf42461f3bcd65 Nov 03 21:16:00.283001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.282931 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-789d48d68-988j4" podUID="50541dcd-b16b-493b-9675-0f232b6abb4e" containerName="console" containerID="cri-o://be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e" gracePeriod=15 Nov 03 21:16:00.503886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.503853 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed2f3a9-343c-4a13-a9d8-68dafc4472fb" path="/var/lib/kubelet/pods/bed2f3a9-343c-4a13-a9d8-68dafc4472fb/volumes" Nov 03 21:16:00.566777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.566721 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:16:00.699362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.699326 5931 generic.go:356] "Generic (PLEG): container finished" podID="cc89e350-e328-41e4-afd4-c55a6d4ae69a" containerID="c030b82ff21a0e25381655c383231ffd76167e32c31b939f29deebffa078d2ea" exitCode=0 Nov 03 21:16:00.699854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.699411 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerDied","Data":"c030b82ff21a0e25381655c383231ffd76167e32c31b939f29deebffa078d2ea"} Nov 03 21:16:00.699854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.699481 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerStarted","Data":"2c038394debb5cc166ec79fdeb4084d4991cdc47656f13549acf42461f3bcd65"} Nov 03 21:16:00.700845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.700821 5931 generic.go:356] "Generic (PLEG): container finished" podID="50541dcd-b16b-493b-9675-0f232b6abb4e" containerID="be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e" exitCode=0 Nov 03 21:16:00.701015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.700869 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789d48d68-988j4" Nov 03 21:16:00.701015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.700887 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789d48d68-988j4" event={"ID":"50541dcd-b16b-493b-9675-0f232b6abb4e","Type":"ContainerDied","Data":"be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e"} Nov 03 21:16:00.701015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.700928 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789d48d68-988j4" event={"ID":"50541dcd-b16b-493b-9675-0f232b6abb4e","Type":"ContainerDied","Data":"9c977e420ca97504dc5628d1a5b563890bf8d3a55c80afc24c7087e56cef7d8e"} Nov 03 21:16:00.701015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.700956 5931 scope.go:122] "RemoveContainer" containerID="be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e" Nov 03 21:16:00.715537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715515 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\") pod \"50541dcd-b16b-493b-9675-0f232b6abb4e\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " Nov 03 21:16:00.715640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715553 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\" (UniqueName: \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\") pod \"50541dcd-b16b-493b-9675-0f232b6abb4e\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " Nov 03 21:16:00.715640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715580 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\") pod \"50541dcd-b16b-493b-9675-0f232b6abb4e\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " Nov 03 21:16:00.715751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715670 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\") pod \"50541dcd-b16b-493b-9675-0f232b6abb4e\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " Nov 03 21:16:00.715751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715726 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\") pod \"50541dcd-b16b-493b-9675-0f232b6abb4e\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " Nov 03 21:16:00.715811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715750 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\") pod \"50541dcd-b16b-493b-9675-0f232b6abb4e\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " Nov 03 21:16:00.715811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715780 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\") pod \"50541dcd-b16b-493b-9675-0f232b6abb4e\" (UID: \"50541dcd-b16b-493b-9675-0f232b6abb4e\") " Nov 03 21:16:00.716373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.715959 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config" pod "50541dcd-b16b-493b-9675-0f232b6abb4e" (UID: "50541dcd-b16b-493b-9675-0f232b6abb4e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:00.716373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.716219 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp" pod "50541dcd-b16b-493b-9675-0f232b6abb4e" (UID: "50541dcd-b16b-493b-9675-0f232b6abb4e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:00.716555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.716530 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert" pod "50541dcd-b16b-493b-9675-0f232b6abb4e" (UID: "50541dcd-b16b-493b-9675-0f232b6abb4e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:00.716744 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.716721 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca" pod "50541dcd-b16b-493b-9675-0f232b6abb4e" (UID: "50541dcd-b16b-493b-9675-0f232b6abb4e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:00.718338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.718314 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj" pod "50541dcd-b16b-493b-9675-0f232b6abb4e" (UID: "50541dcd-b16b-493b-9675-0f232b6abb4e"). InnerVolumeSpecName "kube-api-access-qq9tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:00.718423 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.718314 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config" pod "50541dcd-b16b-493b-9675-0f232b6abb4e" (UID: "50541dcd-b16b-493b-9675-0f232b6abb4e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:00.718669 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.718651 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert" pod "50541dcd-b16b-493b-9675-0f232b6abb4e" (UID: "50541dcd-b16b-493b-9675-0f232b6abb4e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:00.718798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.718777 5931 scope.go:122] "RemoveContainer" containerID="be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e" Nov 03 21:16:00.719094 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:16:00.719079 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e\": container with ID starting with be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e not found: ID does not exist" containerID="be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e" Nov 03 21:16:00.719145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.719103 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e"} err="failed to get container status \"be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e\": rpc error: code = NotFound desc = could not find container \"be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e\": container with ID starting with be11e47089315d1958130f7c91d4cba47af8763ac58e51cddfe1415ed07f439e not found: ID does not exist" Nov 03 21:16:00.817192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.817130 5931 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-console-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:00.817313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.817298 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq9tj\" (UniqueName: \"kubernetes.io/projected/50541dcd-b16b-493b-9675-0f232b6abb4e-kube-api-access-qq9tj\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:00.817453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.817429 5931 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-serving-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:00.817521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.817486 5931 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-oauth-serving-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:00.817521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.817503 5931 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50541dcd-b16b-493b-9675-0f232b6abb4e-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:00.817645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.817518 5931 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50541dcd-b16b-493b-9675-0f232b6abb4e-console-oauth-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:00.817645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:00.817534 5931 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50541dcd-b16b-493b-9675-0f232b6abb4e-tmp\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.132940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.132891 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-789d48d68-988j4"] Nov 03 21:16:01.136838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.136791 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-789d48d68-988j4"] Nov 03 21:16:01.644950 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.644921 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2xkcf" Nov 03 21:16:01.707389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.707346 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerStarted","Data":"e62c07cb2c00487d5db59688695ff1828f3bc07b4204cdaf4624504c3cbe8c7c"} Nov 03 21:16:01.707837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.707422 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerStarted","Data":"cd59e227ab6da2ec2c358e78b7f5c8dc7d3d9e5356283c5426a8dda10be67c6a"} Nov 03 21:16:01.707837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.707469 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerStarted","Data":"ce40ebbc1d2d351a245890cad7ce89880084b508cd2d3d82a5ac1aa44c690585"} Nov 03 21:16:01.707837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.707500 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerStarted","Data":"9d2eee25d4ba8b7b33fb38ad6fb5489921709202ad880a8021f9be4deb587663"} Nov 03 21:16:01.707837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.707531 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerStarted","Data":"648bb6b1a0afc3e0c342a473a780662f361eeee94c07d3a36ea3b95da4914ff1"} Nov 03 21:16:01.707837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.707566 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc89e350-e328-41e4-afd4-c55a6d4ae69a","Type":"ContainerStarted","Data":"7fdc1f9ae011c85357afc94fad1b6fbe3388678bd866122358534590fe383d86"} Nov 03 21:16:01.885515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:01.885451 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.864190841 podStartE2EDuration="3.885429613s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.238772ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:01.88226701 +0000 UTC m=+29.493114821" watchObservedRunningTime="2026-11-03 21:16:01.885429613 +0000 UTC m=+29.496277426" Nov 03 21:16:02.503560 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:02.503527 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50541dcd-b16b-493b-9675-0f232b6abb4e" path="/var/lib/kubelet/pods/50541dcd-b16b-493b-9675-0f232b6abb4e/volumes" Nov 03 21:16:03.066906 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:16:03.066864 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:16:04.272597 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:04.272562 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:16:11.066106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:11.066056 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:11.294338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:11.294305 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:11.543505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:11.543465 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:11.792982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:11.792950 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:21.667051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.666983 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9797677c8-8gnvt" podUID="ec556062-fda8-4a75-aac7-fb519d35285d" containerName="console" containerID="cri-o://394dcc9d384692b839ed50befe02f059de2f8fdfbe6689a3bf137a152e1c0501" gracePeriod=15 Nov 03 21:16:21.785063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.785033 5931 generic.go:356] "Generic (PLEG): container finished" podID="ec556062-fda8-4a75-aac7-fb519d35285d" containerID="394dcc9d384692b839ed50befe02f059de2f8fdfbe6689a3bf137a152e1c0501" exitCode=0 Nov 03 21:16:21.785172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.785112 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9797677c8-8gnvt" event={"ID":"ec556062-fda8-4a75-aac7-fb519d35285d","Type":"ContainerDied","Data":"394dcc9d384692b839ed50befe02f059de2f8fdfbe6689a3bf137a152e1c0501"} Nov 03 21:16:21.919892 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.919843 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:16:21.993338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993311 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993355 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993374 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993403 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\" (UniqueName: \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993441 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993463 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993500 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993545 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\") pod \"ec556062-fda8-4a75-aac7-fb519d35285d\" (UID: \"ec556062-fda8-4a75-aac7-fb519d35285d\") " Nov 03 21:16:21.993771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993591 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:21.993921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993807 5931 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec556062-fda8-4a75-aac7-fb519d35285d-tmp\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:21.993921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993849 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:21.994015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.993924 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:21.994065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.994044 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:21.994141 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.994117 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:21.995923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.995903 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "kube-api-access-wdmw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:21.996188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.996172 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:21.996274 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:21.996253 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert" pod "ec556062-fda8-4a75-aac7-fb519d35285d" (UID: "ec556062-fda8-4a75-aac7-fb519d35285d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:22.094974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.094932 5931 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-oauth-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.094974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.094970 5931 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-console-config\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.094974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.094985 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdmw5\" (UniqueName: \"kubernetes.io/projected/ec556062-fda8-4a75-aac7-fb519d35285d-kube-api-access-wdmw5\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.095199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.094999 5931 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.095199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.095009 5931 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-oauth-serving-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.095199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.095017 5931 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec556062-fda8-4a75-aac7-fb519d35285d-trusted-ca-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.095199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.095026 5931 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec556062-fda8-4a75-aac7-fb519d35285d-console-serving-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.789195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.789122 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9797677c8-8gnvt" Nov 03 21:16:22.789195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.789134 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9797677c8-8gnvt" event={"ID":"ec556062-fda8-4a75-aac7-fb519d35285d","Type":"ContainerDied","Data":"97faeb06d3d26c6530b4be576c4a62a293497cbbe223698ba86ec450df265aef"} Nov 03 21:16:22.789195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.789187 5931 scope.go:122] "RemoveContainer" containerID="394dcc9d384692b839ed50befe02f059de2f8fdfbe6689a3bf137a152e1c0501" Nov 03 21:16:22.949870 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.949829 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9797677c8-8gnvt"] Nov 03 21:16:22.998687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:22.998657 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9797677c8-8gnvt"] Nov 03 21:16:24.502841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:24.502802 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec556062-fda8-4a75-aac7-fb519d35285d" path="/var/lib/kubelet/pods/ec556062-fda8-4a75-aac7-fb519d35285d/volumes" Nov 03 21:16:33.066658 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:16:33.066597 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:16:35.822910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:35.822871 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:56.086546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:56.086507 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:16:56.102338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:56.102310 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:16:56.921857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:16:56.921831 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 21:17:03.066035 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:17:03.065998 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:17:18.336952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:17:18.336868 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:27.211219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:17:27.211178 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:33.067217 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:17:33.067180 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:17:36.213021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:17:36.212990 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:03.065580 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:18:03.065544 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:18:17.639420 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:17.639386 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:17.862033 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:17.862004 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:32.742372 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:32.742336 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:33.004915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:33.004836 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:33.065462 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:18:33.065420 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:18:34.112632 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:34.112576 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:34.432191 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:34.432114 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:35.132945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:35.132913 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:36.506874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:36.506841 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:37.109493 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.109458 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc"] Nov 03 21:18:37.109882 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.109864 5931 cpu_manager.go:414] "containerMap: removing container" podUID="ec556062-fda8-4a75-aac7-fb519d35285d" containerName="console" Nov 03 21:18:37.109946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.109885 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec556062-fda8-4a75-aac7-fb519d35285d" containerName="console" Nov 03 21:18:37.109946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.109910 5931 cpu_manager.go:414] "containerMap: removing container" podUID="50541dcd-b16b-493b-9675-0f232b6abb4e" containerName="console" Nov 03 21:18:37.109946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.109915 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50541dcd-b16b-493b-9675-0f232b6abb4e" containerName="console" Nov 03 21:18:37.110053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.109963 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="50541dcd-b16b-493b-9675-0f232b6abb4e" containerName="console" Nov 03 21:18:37.110053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.109974 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec556062-fda8-4a75-aac7-fb519d35285d" containerName="console" Nov 03 21:18:37.111757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.111742 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.170980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.170944 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.171123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.170998 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbvb\" (UniqueName: \"kubernetes.io/projected/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-kube-api-access-wfbvb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.171180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.171119 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.213128 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.213102 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 21:18:37.213128 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.213102 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 21:18:37.213303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.213260 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-v6s9g\"" Nov 03 21:18:37.213820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.213799 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc"] Nov 03 21:18:37.272178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.272153 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.272333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.272188 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.272333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.272233 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbvb\" (UniqueName: \"kubernetes.io/projected/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-kube-api-access-wfbvb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.272603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.272583 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.272727 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.272602 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.328757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.328732 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbvb\" (UniqueName: \"kubernetes.io/projected/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-kube-api-access-wfbvb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.423484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.423416 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:18:37.790259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.790142 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc"] Nov 03 21:18:37.801120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:37.801097 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:18:38.169187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.169136 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k"] Nov 03 21:18:38.171916 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.171898 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.236517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.236488 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" event={"ID":"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f","Type":"ContainerStarted","Data":"c19aec1007e655235099b0f6f3366ba801a0582851cab395d37ecdf56610edf9"} Nov 03 21:18:38.273129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.273103 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-d74fn\"" Nov 03 21:18:38.273891 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.273871 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k"] Nov 03 21:18:38.274289 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.274266 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297"] Nov 03 21:18:38.276332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.276317 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.281356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.281333 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.281454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.281389 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.281496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.281459 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqst\" (UniqueName: \"kubernetes.io/projected/8040ed3b-194f-43f6-8f57-3393d55c9bd6-kube-api-access-6wqst\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.326383 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.326343 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297"] Nov 03 21:18:38.328218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.328192 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-qrvxn\"" Nov 03 21:18:38.368340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.368314 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6"] Nov 03 21:18:38.370498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.370484 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.382417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382396 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvlz5\" (UniqueName: \"kubernetes.io/projected/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-kube-api-access-kvlz5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.382501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382443 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.382501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382464 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.382655 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382638 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.382703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382674 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.382759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382743 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqst\" (UniqueName: \"kubernetes.io/projected/8040ed3b-194f-43f6-8f57-3393d55c9bd6-kube-api-access-6wqst\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.382942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382923 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.382976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.382942 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.422090 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.422043 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mtw5j\"" Nov 03 21:18:38.428481 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.428452 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6"] Nov 03 21:18:38.439106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.439083 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqst\" (UniqueName: \"kubernetes.io/projected/8040ed3b-194f-43f6-8f57-3393d55c9bd6-kube-api-access-6wqst\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.468200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.468163 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm"] Nov 03 21:18:38.470512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.470498 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.483770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.483752 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:18:38.483963 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.483945 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.484000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.483979 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.484045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.484013 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvlz5\" (UniqueName: \"kubernetes.io/projected/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-kube-api-access-kvlz5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.484301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.484280 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.484371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.484330 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.484423 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.484389 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsv9w\" (UniqueName: \"kubernetes.io/projected/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-kube-api-access-fsv9w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.484595 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.484581 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.484683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.484667 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.522167 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.522134 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9lq66\"" Nov 03 21:18:38.534361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.534338 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm"] Nov 03 21:18:38.546642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.541241 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvlz5\" (UniqueName: \"kubernetes.io/projected/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-kube-api-access-kvlz5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.585260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585228 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.585260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585264 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.585474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585391 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.585534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585511 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.585664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585599 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsv9w\" (UniqueName: \"kubernetes.io/projected/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-kube-api-access-fsv9w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.585664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585634 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.585765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585676 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.585765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.585688 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtcd\" (UniqueName: \"kubernetes.io/projected/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-kube-api-access-2qtcd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.588181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.588166 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:18:38.644809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.644785 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsv9w\" (UniqueName: \"kubernetes.io/projected/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-kube-api-access-fsv9w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.681421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.681362 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:18:38.686554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.686531 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.686630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.686597 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.686718 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.686698 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtcd\" (UniqueName: \"kubernetes.io/projected/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-kube-api-access-2qtcd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.686917 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.686898 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.686961 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.686915 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.743136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.743107 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtcd\" (UniqueName: \"kubernetes.io/projected/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-kube-api-access-2qtcd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.782419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.782391 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:18:38.852139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.852041 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k"] Nov 03 21:18:38.859786 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:18:38.859742 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8040ed3b_194f_43f6_8f57_3393d55c9bd6.slice/crio-6111fa7551df6e7d312ec3060937d51c25af2179f8cca1c268df6cb24299fb5d WatchSource:0}: Error finding container 6111fa7551df6e7d312ec3060937d51c25af2179f8cca1c268df6cb24299fb5d: Status 404 returned error can't find the container with id 6111fa7551df6e7d312ec3060937d51c25af2179f8cca1c268df6cb24299fb5d Nov 03 21:18:38.956838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:38.956722 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297"] Nov 03 21:18:38.961730 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:18:38.961704 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fef1d6_e967_4d1f_bf8d_2a0e8f9b0b93.slice/crio-54ccdc295a33c47839084ba3ea1789ec83d46b116a5596f7843db20ffcc84758 WatchSource:0}: Error finding container 54ccdc295a33c47839084ba3ea1789ec83d46b116a5596f7843db20ffcc84758: Status 404 returned error can't find the container with id 54ccdc295a33c47839084ba3ea1789ec83d46b116a5596f7843db20ffcc84758 Nov 03 21:18:39.045734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:39.045687 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6"] Nov 03 21:18:39.049562 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:18:39.049532 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8467049f_d0a4_4f96_9fd8_78f3c8ead4d9.slice/crio-dfa3f2105506f46e9dc7712fd2b575513fbae9c49e55bd51882810a849a52f37 WatchSource:0}: Error finding container dfa3f2105506f46e9dc7712fd2b575513fbae9c49e55bd51882810a849a52f37: Status 404 returned error can't find the container with id dfa3f2105506f46e9dc7712fd2b575513fbae9c49e55bd51882810a849a52f37 Nov 03 21:18:39.156692 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:39.156641 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm"] Nov 03 21:18:39.160271 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:18:39.160242 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560cfa84_2d4d_4c0d_8fdc_fbea8447288a.slice/crio-fa3280d1edf9eba06ec59b975500d879e18ed254d9f9d18eb77d38850986292e WatchSource:0}: Error finding container fa3280d1edf9eba06ec59b975500d879e18ed254d9f9d18eb77d38850986292e: Status 404 returned error can't find the container with id fa3280d1edf9eba06ec59b975500d879e18ed254d9f9d18eb77d38850986292e Nov 03 21:18:39.242082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:39.242003 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" event={"ID":"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9","Type":"ContainerStarted","Data":"dfa3f2105506f46e9dc7712fd2b575513fbae9c49e55bd51882810a849a52f37"} Nov 03 21:18:39.243270 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:39.243242 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" event={"ID":"8040ed3b-194f-43f6-8f57-3393d55c9bd6","Type":"ContainerStarted","Data":"6111fa7551df6e7d312ec3060937d51c25af2179f8cca1c268df6cb24299fb5d"} Nov 03 21:18:39.244505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:39.244480 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" event={"ID":"560cfa84-2d4d-4c0d-8fdc-fbea8447288a","Type":"ContainerStarted","Data":"fa3280d1edf9eba06ec59b975500d879e18ed254d9f9d18eb77d38850986292e"} Nov 03 21:18:39.245691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:39.245662 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" event={"ID":"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93","Type":"ContainerStarted","Data":"54ccdc295a33c47839084ba3ea1789ec83d46b116a5596f7843db20ffcc84758"} Nov 03 21:18:40.217868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.216081 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m"] Nov 03 21:18:40.219157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.219091 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.301737 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.301689 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.301921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.301872 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.301986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.301939 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvgb6\" (UniqueName: \"kubernetes.io/projected/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-kube-api-access-wvgb6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.321660 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.321629 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m"] Nov 03 21:18:40.321826 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.321789 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-xhsmb\"" Nov 03 21:18:40.402702 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.402654 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.402888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.402742 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvgb6\" (UniqueName: \"kubernetes.io/projected/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-kube-api-access-wvgb6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.402888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.402798 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.403320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.403295 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.403455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.403304 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.409952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.409920 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:40.463927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.463864 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvgb6\" (UniqueName: \"kubernetes.io/projected/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-kube-api-access-wvgb6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.536343 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.536266 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:18:40.655211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.655145 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:40.965636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:40.965582 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m"] Nov 03 21:18:40.976471 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:18:40.976441 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69661b44_d44b_4d3d_a2f7_bed0f91cb9fe.slice/crio-f90dea9650292459ad6beb93c955e6a2ed9c865c6b6b807148815135d96dabea WatchSource:0}: Error finding container f90dea9650292459ad6beb93c955e6a2ed9c865c6b6b807148815135d96dabea: Status 404 returned error can't find the container with id f90dea9650292459ad6beb93c955e6a2ed9c865c6b6b807148815135d96dabea Nov 03 21:18:41.033525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.031576 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f"] Nov 03 21:18:41.036016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.035988 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.090335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.090299 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2l2wp\"" Nov 03 21:18:41.091670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.091646 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f"] Nov 03 21:18:41.112909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.112874 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.113069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.112944 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.113069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.112999 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48l8d\" (UniqueName: \"kubernetes.io/projected/640959cb-a171-4844-b3c0-148662afbd40-kube-api-access-48l8d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.214273 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.214232 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48l8d\" (UniqueName: \"kubernetes.io/projected/640959cb-a171-4844-b3c0-148662afbd40-kube-api-access-48l8d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.214459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.214317 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.214459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.214376 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.214867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.214840 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.215436 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.215411 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.267249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.267067 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" event={"ID":"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe","Type":"ContainerStarted","Data":"f90dea9650292459ad6beb93c955e6a2ed9c865c6b6b807148815135d96dabea"} Nov 03 21:18:41.273923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.273894 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:41.275952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.275926 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48l8d\" (UniqueName: \"kubernetes.io/projected/640959cb-a171-4844-b3c0-148662afbd40-kube-api-access-48l8d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.359953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.359757 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:18:41.505973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.505937 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:41.781472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:41.781389 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f"] Nov 03 21:18:42.278594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:42.278516 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" event={"ID":"640959cb-a171-4844-b3c0-148662afbd40","Type":"ContainerStarted","Data":"53646f5549dbf705907bba48bce8101f89e70021209f1920f353b595e1dbba03"} Nov 03 21:18:44.632287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:44.632203 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:47.412743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:47.412653 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:50.351091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:50.351048 5931 generic.go:356] "Generic (PLEG): container finished" podID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerID="ebb9f4d9db5dc6b615d5eb9ac6fadc35c2d4a86f73600b77b5db812da07fc4ae" exitCode=0 Nov 03 21:18:50.351668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:50.351151 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" event={"ID":"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f","Type":"ContainerDied","Data":"ebb9f4d9db5dc6b615d5eb9ac6fadc35c2d4a86f73600b77b5db812da07fc4ae"} Nov 03 21:18:50.361741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:50.361656 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:50.583945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:50.583805 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:52.362919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.362884 5931 generic.go:356] "Generic (PLEG): container finished" podID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerID="5917ff67ed9d887ac1600a4792bc776bcbf33889d01b31e85018b302993c04a0" exitCode=0 Nov 03 21:18:52.363390 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.362976 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" event={"ID":"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93","Type":"ContainerDied","Data":"5917ff67ed9d887ac1600a4792bc776bcbf33889d01b31e85018b302993c04a0"} Nov 03 21:18:52.365324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.365296 5931 generic.go:356] "Generic (PLEG): container finished" podID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerID="3a6b06373c260d53df50487a6cb873b5bbc143459eb1ec08ccb562906c8cb3e3" exitCode=0 Nov 03 21:18:52.365443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.365363 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" event={"ID":"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f","Type":"ContainerDied","Data":"3a6b06373c260d53df50487a6cb873b5bbc143459eb1ec08ccb562906c8cb3e3"} Nov 03 21:18:52.367049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.367024 5931 generic.go:356] "Generic (PLEG): container finished" podID="640959cb-a171-4844-b3c0-148662afbd40" containerID="39e4fe003d82e23cb752e490a12af02a7086e312fba72071c6021b3c9e207dcc" exitCode=0 Nov 03 21:18:52.367150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.367132 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" event={"ID":"640959cb-a171-4844-b3c0-148662afbd40","Type":"ContainerDied","Data":"39e4fe003d82e23cb752e490a12af02a7086e312fba72071c6021b3c9e207dcc"} Nov 03 21:18:52.369873 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.369850 5931 generic.go:356] "Generic (PLEG): container finished" podID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerID="f25be0b72bfb89cbec3708cac80a0461b4a31957992c24619295b26409e1795e" exitCode=0 Nov 03 21:18:52.369976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.369948 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" event={"ID":"8040ed3b-194f-43f6-8f57-3393d55c9bd6","Type":"ContainerDied","Data":"f25be0b72bfb89cbec3708cac80a0461b4a31957992c24619295b26409e1795e"} Nov 03 21:18:52.371962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.371940 5931 generic.go:356] "Generic (PLEG): container finished" podID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerID="ce7d4d5ac697dba079e31ab48004a5d316bd0a48ccdb5737cea3a6d92920e5ae" exitCode=0 Nov 03 21:18:52.372046 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:52.371970 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" event={"ID":"560cfa84-2d4d-4c0d-8fdc-fbea8447288a","Type":"ContainerDied","Data":"ce7d4d5ac697dba079e31ab48004a5d316bd0a48ccdb5737cea3a6d92920e5ae"} Nov 03 21:18:53.377685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:53.377643 5931 generic.go:356] "Generic (PLEG): container finished" podID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerID="d41c848c7e1d94b9f00af23697152374af3020d09bbc830f5f7985f5b0b6fc0f" exitCode=0 Nov 03 21:18:53.378155 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:53.377727 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" event={"ID":"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe","Type":"ContainerDied","Data":"d41c848c7e1d94b9f00af23697152374af3020d09bbc830f5f7985f5b0b6fc0f"} Nov 03 21:18:53.379821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:53.379724 5931 generic.go:356] "Generic (PLEG): container finished" podID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerID="0feece073e25f1855e9dc4bced4fc978c2dd3858bd996c5d5f123483f24af695" exitCode=0 Nov 03 21:18:53.379821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:53.379769 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" event={"ID":"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9","Type":"ContainerDied","Data":"0feece073e25f1855e9dc4bced4fc978c2dd3858bd996c5d5f123483f24af695"} Nov 03 21:18:54.387692 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:54.387045 5931 generic.go:356] "Generic (PLEG): container finished" podID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerID="ae99d4ed077b058688001e742e657a6f22b19e975ff94caa7a0700b1d1af758e" exitCode=0 Nov 03 21:18:54.387692 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:54.387158 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" event={"ID":"560cfa84-2d4d-4c0d-8fdc-fbea8447288a","Type":"ContainerDied","Data":"ae99d4ed077b058688001e742e657a6f22b19e975ff94caa7a0700b1d1af758e"} Nov 03 21:18:55.395363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:55.395088 5931 generic.go:356] "Generic (PLEG): container finished" podID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerID="54c9d3a110eee952be46bf2aa27a5a403e95a9c15efbec6dfcc4699378a0f830" exitCode=0 Nov 03 21:18:55.395363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:55.395246 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" event={"ID":"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93","Type":"ContainerDied","Data":"54c9d3a110eee952be46bf2aa27a5a403e95a9c15efbec6dfcc4699378a0f830"} Nov 03 21:18:55.399417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:55.398717 5931 generic.go:356] "Generic (PLEG): container finished" podID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerID="18772ce0895aa621b15d2f5fb59923abf1e95576251c1fa3d83993c7ef332fed" exitCode=0 Nov 03 21:18:55.399417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:55.398790 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" event={"ID":"8040ed3b-194f-43f6-8f57-3393d55c9bd6","Type":"ContainerDied","Data":"18772ce0895aa621b15d2f5fb59923abf1e95576251c1fa3d83993c7ef332fed"} Nov 03 21:18:56.407249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:56.407210 5931 generic.go:356] "Generic (PLEG): container finished" podID="640959cb-a171-4844-b3c0-148662afbd40" containerID="14717d8685620b0078007e3dc50e8ee3be77ef953d1e16522cca836d0985d3f5" exitCode=0 Nov 03 21:18:56.407695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:56.407308 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" event={"ID":"640959cb-a171-4844-b3c0-148662afbd40","Type":"ContainerDied","Data":"14717d8685620b0078007e3dc50e8ee3be77ef953d1e16522cca836d0985d3f5"} Nov 03 21:18:57.416141 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:57.416104 5931 generic.go:356] "Generic (PLEG): container finished" podID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerID="e2d2c05ab24264e4a20e031ec317a03b1fb42c2582a72224c07cbbec71cbc209" exitCode=0 Nov 03 21:18:57.416706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:18:57.416196 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" event={"ID":"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe","Type":"ContainerDied","Data":"e2d2c05ab24264e4a20e031ec317a03b1fb42c2582a72224c07cbbec71cbc209"} Nov 03 21:19:03.080202 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:19:03.080116 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:19:03.456680 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:03.456625 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" event={"ID":"8040ed3b-194f-43f6-8f57-3393d55c9bd6","Type":"ContainerStarted","Data":"f0cb43a968ce599e2e1ce7959388327bf6bc0511db28edfaefa83a6fafe38478"} Nov 03 21:19:03.459472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:03.459442 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" event={"ID":"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93","Type":"ContainerStarted","Data":"5abd6a2623d3ae1db3e76a0d5120b2e35530fff9bbf683d50ce1e644d897d733"} Nov 03 21:19:03.462712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:03.462582 5931 generic.go:356] "Generic (PLEG): container finished" podID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerID="75ad6a8ea335daa56dbbcb7573473bfd1e6109a092a0f5b170c0f1ea4b784b80" exitCode=0 Nov 03 21:19:03.462712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:03.462664 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" event={"ID":"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f","Type":"ContainerDied","Data":"75ad6a8ea335daa56dbbcb7573473bfd1e6109a092a0f5b170c0f1ea4b784b80"} Nov 03 21:19:03.578320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:03.578258 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" podStartSLOduration=2.312166963 podStartE2EDuration="25.578240719s" totalImagesPullingTime="23.266073756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:03.577285483 +0000 UTC m=+211.188133298" watchObservedRunningTime="2026-11-03 21:19:03.578240719 +0000 UTC m=+211.189088536" Nov 03 21:19:03.691230 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:03.691182 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" podStartSLOduration=3.011362028 podStartE2EDuration="25.691168153s" totalImagesPullingTime="22.679806125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:03.690041379 +0000 UTC m=+211.300889393" watchObservedRunningTime="2026-11-03 21:19:03.691168153 +0000 UTC m=+211.302015968" Nov 03 21:19:04.469575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.469534 5931 generic.go:356] "Generic (PLEG): container finished" podID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerID="5abd6a2623d3ae1db3e76a0d5120b2e35530fff9bbf683d50ce1e644d897d733" exitCode=0 Nov 03 21:19:04.470047 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.469678 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" event={"ID":"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93","Type":"ContainerDied","Data":"5abd6a2623d3ae1db3e76a0d5120b2e35530fff9bbf683d50ce1e644d897d733"} Nov 03 21:19:04.472241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.472205 5931 generic.go:356] "Generic (PLEG): container finished" podID="640959cb-a171-4844-b3c0-148662afbd40" containerID="7bdbe00cfdd0c7e0f20dfdf9b0d77bdd1828a80d683d61afb625f799e97b47b5" exitCode=0 Nov 03 21:19:04.472348 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.472238 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" event={"ID":"640959cb-a171-4844-b3c0-148662afbd40","Type":"ContainerDied","Data":"7bdbe00cfdd0c7e0f20dfdf9b0d77bdd1828a80d683d61afb625f799e97b47b5"} Nov 03 21:19:04.475070 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.475047 5931 generic.go:356] "Generic (PLEG): container finished" podID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerID="f0cb43a968ce599e2e1ce7959388327bf6bc0511db28edfaefa83a6fafe38478" exitCode=0 Nov 03 21:19:04.475163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.475137 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" event={"ID":"8040ed3b-194f-43f6-8f57-3393d55c9bd6","Type":"ContainerDied","Data":"f0cb43a968ce599e2e1ce7959388327bf6bc0511db28edfaefa83a6fafe38478"} Nov 03 21:19:04.618193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.618162 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:19:04.680232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.680202 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-kube-api-access-wfbvb\" (UniqueName: \"kubernetes.io/projected/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-kube-api-access-wfbvb\") pod \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " Nov 03 21:19:04.680442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.680257 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-bundle\") pod \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " Nov 03 21:19:04.680442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.680304 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-util\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-util\") pod \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\" (UID: \"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f\") " Nov 03 21:19:04.681872 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.681835 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-bundle" pod "0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" (UID: "0a401ffa-dd28-42ca-9c3e-5b2f527dd95f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:04.683330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.683303 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-kube-api-access-wfbvb" pod "0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" (UID: "0a401ffa-dd28-42ca-9c3e-5b2f527dd95f"). InnerVolumeSpecName "kube-api-access-wfbvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:04.686180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.686154 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-util" pod "0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" (UID: "0a401ffa-dd28-42ca-9c3e-5b2f527dd95f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:04.781471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.781439 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbvb\" (UniqueName: \"kubernetes.io/projected/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-kube-api-access-wfbvb\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:04.781471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.781473 5931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:04.781694 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:04.781483 5931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a401ffa-dd28-42ca-9c3e-5b2f527dd95f-util\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.481598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.481558 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" event={"ID":"0a401ffa-dd28-42ca-9c3e-5b2f527dd95f","Type":"ContainerDied","Data":"c19aec1007e655235099b0f6f3366ba801a0582851cab395d37ecdf56610edf9"} Nov 03 21:19:05.481598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.481592 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kpjhc" Nov 03 21:19:05.482037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.481641 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19aec1007e655235099b0f6f3366ba801a0582851cab395d37ecdf56610edf9" Nov 03 21:19:05.486238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.486173 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" event={"ID":"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe","Type":"ContainerStarted","Data":"0dc85072ed409825274956d41ee13596d9fd434b25dbdfd91d9291931ee6cc5f"} Nov 03 21:19:05.613131 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.613078 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" podStartSLOduration=1.833962503 podStartE2EDuration="25.613060037s" totalImagesPullingTime="23.779097534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:05.612768766 +0000 UTC m=+213.223616581" watchObservedRunningTime="2026-11-03 21:19:05.613060037 +0000 UTC m=+213.223907853" Nov 03 21:19:05.628627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.628595 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:19:05.651464 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.651444 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:19:05.656764 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.656747 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:19:05.688528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688506 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-kube-api-access-kvlz5\" (UniqueName: \"kubernetes.io/projected/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-kube-api-access-kvlz5\") pod \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " Nov 03 21:19:05.688658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688548 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-bundle\") pod \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " Nov 03 21:19:05.688658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688580 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640959cb-a171-4844-b3c0-148662afbd40-kube-api-access-48l8d\" (UniqueName: \"kubernetes.io/projected/640959cb-a171-4844-b3c0-148662afbd40-kube-api-access-48l8d\") pod \"640959cb-a171-4844-b3c0-148662afbd40\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " Nov 03 21:19:05.688658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688626 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-util\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-util\") pod \"640959cb-a171-4844-b3c0-148662afbd40\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " Nov 03 21:19:05.688796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688661 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-util\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-util\") pod \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\" (UID: \"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93\") " Nov 03 21:19:05.688796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688709 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-util\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-util\") pod \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " Nov 03 21:19:05.688796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688737 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-bundle\") pod \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " Nov 03 21:19:05.688796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688771 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-bundle\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-bundle\") pod \"640959cb-a171-4844-b3c0-148662afbd40\" (UID: \"640959cb-a171-4844-b3c0-148662afbd40\") " Nov 03 21:19:05.688985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.688811 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8040ed3b-194f-43f6-8f57-3393d55c9bd6-kube-api-access-6wqst\" (UniqueName: \"kubernetes.io/projected/8040ed3b-194f-43f6-8f57-3393d55c9bd6-kube-api-access-6wqst\") pod \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\" (UID: \"8040ed3b-194f-43f6-8f57-3393d55c9bd6\") " Nov 03 21:19:05.689235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.689184 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-bundle" pod "c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" (UID: "c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:05.691021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.690023 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-bundle" pod "8040ed3b-194f-43f6-8f57-3393d55c9bd6" (UID: "8040ed3b-194f-43f6-8f57-3393d55c9bd6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:05.691021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.690047 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-bundle" pod "640959cb-a171-4844-b3c0-148662afbd40" (UID: "640959cb-a171-4844-b3c0-148662afbd40"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:05.691021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.690980 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640959cb-a171-4844-b3c0-148662afbd40-kube-api-access-48l8d" pod "640959cb-a171-4844-b3c0-148662afbd40" (UID: "640959cb-a171-4844-b3c0-148662afbd40"). InnerVolumeSpecName "kube-api-access-48l8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:05.691302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.691281 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-kube-api-access-kvlz5" pod "c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" (UID: "c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93"). InnerVolumeSpecName "kube-api-access-kvlz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:05.691456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.691438 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8040ed3b-194f-43f6-8f57-3393d55c9bd6-kube-api-access-6wqst" pod "8040ed3b-194f-43f6-8f57-3393d55c9bd6" (UID: "8040ed3b-194f-43f6-8f57-3393d55c9bd6"). InnerVolumeSpecName "kube-api-access-6wqst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:05.694724 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.694697 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-util" pod "c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" (UID: "c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:05.695100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.695080 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-util" pod "640959cb-a171-4844-b3c0-148662afbd40" (UID: "640959cb-a171-4844-b3c0-148662afbd40"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:05.695624 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.695591 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-util" pod "8040ed3b-194f-43f6-8f57-3393d55c9bd6" (UID: "8040ed3b-194f-43f6-8f57-3393d55c9bd6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790185 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqst\" (UniqueName: \"kubernetes.io/projected/8040ed3b-194f-43f6-8f57-3393d55c9bd6-kube-api-access-6wqst\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790220 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvlz5\" (UniqueName: \"kubernetes.io/projected/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-kube-api-access-kvlz5\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790236 5931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790252 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48l8d\" (UniqueName: \"kubernetes.io/projected/640959cb-a171-4844-b3c0-148662afbd40-kube-api-access-48l8d\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790267 5931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-util\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790280 5931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93-util\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790293 5931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-util\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790305 5931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8040ed3b-194f-43f6-8f57-3393d55c9bd6-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:05.790334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:05.790316 5931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/640959cb-a171-4844-b3c0-148662afbd40-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:06.492093 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.492059 5931 generic.go:356] "Generic (PLEG): container finished" podID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerID="0dc85072ed409825274956d41ee13596d9fd434b25dbdfd91d9291931ee6cc5f" exitCode=0 Nov 03 21:19:06.492527 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.492142 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" event={"ID":"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe","Type":"ContainerDied","Data":"0dc85072ed409825274956d41ee13596d9fd434b25dbdfd91d9291931ee6cc5f"} Nov 03 21:19:06.494007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.493988 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" Nov 03 21:19:06.494119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.494004 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czdz8f" event={"ID":"640959cb-a171-4844-b3c0-148662afbd40","Type":"ContainerDied","Data":"53646f5549dbf705907bba48bce8101f89e70021209f1920f353b595e1dbba03"} Nov 03 21:19:06.494119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.494047 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53646f5549dbf705907bba48bce8101f89e70021209f1920f353b595e1dbba03" Nov 03 21:19:06.495875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.495850 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" Nov 03 21:19:06.495875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.495855 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65q78k" event={"ID":"8040ed3b-194f-43f6-8f57-3393d55c9bd6","Type":"ContainerDied","Data":"6111fa7551df6e7d312ec3060937d51c25af2179f8cca1c268df6cb24299fb5d"} Nov 03 21:19:06.496007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.495894 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6111fa7551df6e7d312ec3060937d51c25af2179f8cca1c268df6cb24299fb5d" Nov 03 21:19:06.498034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.498013 5931 generic.go:356] "Generic (PLEG): container finished" podID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerID="ebcb219dbf13be58aaa0dc579051d2197120baaece575690fef5227f9369e3e8" exitCode=0 Nov 03 21:19:06.498120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.498083 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" event={"ID":"560cfa84-2d4d-4c0d-8fdc-fbea8447288a","Type":"ContainerDied","Data":"ebcb219dbf13be58aaa0dc579051d2197120baaece575690fef5227f9369e3e8"} Nov 03 21:19:06.500199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.500182 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" Nov 03 21:19:06.504097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.504074 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfh297" event={"ID":"c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93","Type":"ContainerDied","Data":"54ccdc295a33c47839084ba3ea1789ec83d46b116a5596f7843db20ffcc84758"} Nov 03 21:19:06.504199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:06.504113 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54ccdc295a33c47839084ba3ea1789ec83d46b116a5596f7843db20ffcc84758" Nov 03 21:19:07.211041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.211013 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:07.506706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.506629 5931 generic.go:356] "Generic (PLEG): container finished" podID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerID="24fcfb79d32e4abae150ab19eb739544f110f6987a197dd0e8a6085d35bc929a" exitCode=0 Nov 03 21:19:07.507051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.506701 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" event={"ID":"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9","Type":"ContainerDied","Data":"24fcfb79d32e4abae150ab19eb739544f110f6987a197dd0e8a6085d35bc929a"} Nov 03 21:19:07.636863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.636836 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:19:07.673401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.673377 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:19:07.707299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.707273 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-util\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-util\") pod \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " Nov 03 21:19:07.707422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.707320 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-bundle\") pod \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " Nov 03 21:19:07.707422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.707341 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-kube-api-access-2qtcd\" (UniqueName: \"kubernetes.io/projected/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-kube-api-access-2qtcd\") pod \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " Nov 03 21:19:07.707422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.707363 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-util\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-util\") pod \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\" (UID: \"560cfa84-2d4d-4c0d-8fdc-fbea8447288a\") " Nov 03 21:19:07.707422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.707387 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-bundle\") pod \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " Nov 03 21:19:07.707422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.707405 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-kube-api-access-wvgb6\" (UniqueName: \"kubernetes.io/projected/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-kube-api-access-wvgb6\") pod \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\" (UID: \"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe\") " Nov 03 21:19:07.707859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.707787 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-bundle" pod "69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" (UID: "69661b44-d44b-4d3d-a2f7-bed0f91cb9fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:07.708209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.708178 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-bundle" pod "560cfa84-2d4d-4c0d-8fdc-fbea8447288a" (UID: "560cfa84-2d4d-4c0d-8fdc-fbea8447288a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:07.709635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.709597 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-kube-api-access-wvgb6" pod "69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" (UID: "69661b44-d44b-4d3d-a2f7-bed0f91cb9fe"). InnerVolumeSpecName "kube-api-access-wvgb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:07.709799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.709770 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-kube-api-access-2qtcd" pod "560cfa84-2d4d-4c0d-8fdc-fbea8447288a" (UID: "560cfa84-2d4d-4c0d-8fdc-fbea8447288a"). InnerVolumeSpecName "kube-api-access-2qtcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:07.713526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.713508 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-util" pod "69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" (UID: "69661b44-d44b-4d3d-a2f7-bed0f91cb9fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:07.715301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.715278 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-util" pod "560cfa84-2d4d-4c0d-8fdc-fbea8447288a" (UID: "560cfa84-2d4d-4c0d-8fdc-fbea8447288a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:07.809188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.809125 5931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-util\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.809188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.809148 5931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.809188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.809158 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtcd\" (UniqueName: \"kubernetes.io/projected/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-kube-api-access-2qtcd\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.809188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.809167 5931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560cfa84-2d4d-4c0d-8fdc-fbea8447288a-util\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.809188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.809176 5931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:07.809188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:07.809183 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvgb6\" (UniqueName: \"kubernetes.io/projected/69661b44-d44b-4d3d-a2f7-bed0f91cb9fe-kube-api-access-wvgb6\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:08.514585 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.514547 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" event={"ID":"69661b44-d44b-4d3d-a2f7-bed0f91cb9fe","Type":"ContainerDied","Data":"f90dea9650292459ad6beb93c955e6a2ed9c865c6b6b807148815135d96dabea"} Nov 03 21:19:08.514971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.514670 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90dea9650292459ad6beb93c955e6a2ed9c865c6b6b807148815135d96dabea" Nov 03 21:19:08.514971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.514800 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh95m" Nov 03 21:19:08.519315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.517661 5931 generic.go:356] "Generic (PLEG): container finished" podID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerID="408e481e9586e4b71f20afdc5f7cdf2032f88a996febd744da9b8b78c19bd1cc" exitCode=0 Nov 03 21:19:08.519315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.517778 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" event={"ID":"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9","Type":"ContainerDied","Data":"408e481e9586e4b71f20afdc5f7cdf2032f88a996febd744da9b8b78c19bd1cc"} Nov 03 21:19:08.520406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.520379 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" event={"ID":"560cfa84-2d4d-4c0d-8fdc-fbea8447288a","Type":"ContainerDied","Data":"fa3280d1edf9eba06ec59b975500d879e18ed254d9f9d18eb77d38850986292e"} Nov 03 21:19:08.520506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.520433 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa3280d1edf9eba06ec59b975500d879e18ed254d9f9d18eb77d38850986292e" Nov 03 21:19:08.520742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:08.520536 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lj5nm" Nov 03 21:19:09.135157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.135127 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:09.401903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.401829 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2"] Nov 03 21:19:09.402140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402127 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerName="util" Nov 03 21:19:09.402175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402144 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerName="util" Nov 03 21:19:09.402175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402155 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerName="pull" Nov 03 21:19:09.402175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402160 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerName="pull" Nov 03 21:19:09.402175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402169 5931 cpu_manager.go:414] "containerMap: removing container" podUID="640959cb-a171-4844-b3c0-148662afbd40" containerName="pull" Nov 03 21:19:09.402175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402173 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640959cb-a171-4844-b3c0-148662afbd40" containerName="pull" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402180 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402187 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402192 5931 cpu_manager.go:414] "containerMap: removing container" podUID="640959cb-a171-4844-b3c0-148662afbd40" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402197 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640959cb-a171-4844-b3c0-148662afbd40" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402205 5931 cpu_manager.go:414] "containerMap: removing container" podUID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402209 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402217 5931 cpu_manager.go:414] "containerMap: removing container" podUID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402223 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402233 5931 cpu_manager.go:414] "containerMap: removing container" podUID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402237 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402244 5931 cpu_manager.go:414] "containerMap: removing container" podUID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerName="pull" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402248 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerName="pull" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402253 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402257 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerName="extract" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402270 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402273 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402278 5931 cpu_manager.go:414] "containerMap: removing container" podUID="640959cb-a171-4844-b3c0-148662afbd40" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402281 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640959cb-a171-4844-b3c0-148662afbd40" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402288 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402292 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerName="util" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402302 5931 cpu_manager.go:414] "containerMap: removing container" podUID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerName="pull" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402309 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerName="pull" Nov 03 21:19:09.402314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402321 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerName="pull" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402325 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerName="pull" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402333 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerName="pull" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402338 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerName="pull" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402344 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402347 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402353 5931 cpu_manager.go:414] "containerMap: removing container" podUID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402357 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402407 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8040ed3b-194f-43f6-8f57-3393d55c9bd6" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402414 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="640959cb-a171-4844-b3c0-148662afbd40" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402418 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="560cfa84-2d4d-4c0d-8fdc-fbea8447288a" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402424 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a401ffa-dd28-42ca-9c3e-5b2f527dd95f" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402433 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fef1d6-e967-4d1f-bf8d-2a0e8f9b0b93" containerName="extract" Nov 03 21:19:09.402929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.402437 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="69661b44-d44b-4d3d-a2f7-bed0f91cb9fe" containerName="extract" Nov 03 21:19:09.404589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.404573 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.458634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.458587 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9s2bq\"" Nov 03 21:19:09.458846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.458832 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:19:09.506996 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.506976 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 21:19:09.508394 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.508375 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2"] Nov 03 21:19:09.525218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.525193 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62bf39a0-1fce-42c4-82cd-2ef6b897ccb6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nzct2\" (UID: \"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.525591 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.525275 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grt5c\" (UniqueName: \"kubernetes.io/projected/62bf39a0-1fce-42c4-82cd-2ef6b897ccb6-kube-api-access-grt5c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nzct2\" (UID: \"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.625839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.625820 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62bf39a0-1fce-42c4-82cd-2ef6b897ccb6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nzct2\" (UID: \"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.626521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.626209 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grt5c\" (UniqueName: \"kubernetes.io/projected/62bf39a0-1fce-42c4-82cd-2ef6b897ccb6-kube-api-access-grt5c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nzct2\" (UID: \"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.626521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.626239 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62bf39a0-1fce-42c4-82cd-2ef6b897ccb6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nzct2\" (UID: \"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.637851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.637832 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:19:09.687221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.687169 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grt5c\" (UniqueName: \"kubernetes.io/projected/62bf39a0-1fce-42c4-82cd-2ef6b897ccb6-kube-api-access-grt5c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nzct2\" (UID: \"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.714177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.714154 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" Nov 03 21:19:09.726995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.726973 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-util\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-util\") pod \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " Nov 03 21:19:09.727091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.727023 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-bundle\") pod \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " Nov 03 21:19:09.727091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.727059 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-kube-api-access-fsv9w\" (UniqueName: \"kubernetes.io/projected/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-kube-api-access-fsv9w\") pod \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\" (UID: \"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9\") " Nov 03 21:19:09.727667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.727646 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-bundle" pod "8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" (UID: "8467049f-d0a4-4f96-9fd8-78f3c8ead4d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:09.729964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.729939 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-kube-api-access-fsv9w" pod "8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" (UID: "8467049f-d0a4-4f96-9fd8-78f3c8ead4d9"). InnerVolumeSpecName "kube-api-access-fsv9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:09.732020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.731999 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-util" pod "8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" (UID: "8467049f-d0a4-4f96-9fd8-78f3c8ead4d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:09.828287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.828254 5931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-util\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:09.828287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.828279 5931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-bundle\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:09.828287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:09.828289 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsv9w\" (UniqueName: \"kubernetes.io/projected/8467049f-d0a4-4f96-9fd8-78f3c8ead4d9-kube-api-access-fsv9w\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:10.137009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:10.136977 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2"] Nov 03 21:19:10.140537 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:10.140506 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bf39a0_1fce_42c4_82cd_2ef6b897ccb6.slice/crio-9a7c824d5d2b2e49bf90422b345194cae5558a339ff7ee2aad9763435877831b WatchSource:0}: Error finding container 9a7c824d5d2b2e49bf90422b345194cae5558a339ff7ee2aad9763435877831b: Status 404 returned error can't find the container with id 9a7c824d5d2b2e49bf90422b345194cae5558a339ff7ee2aad9763435877831b Nov 03 21:19:10.529644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:10.529597 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" Nov 03 21:19:10.529644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:10.529632 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lnk6" event={"ID":"8467049f-d0a4-4f96-9fd8-78f3c8ead4d9","Type":"ContainerDied","Data":"dfa3f2105506f46e9dc7712fd2b575513fbae9c49e55bd51882810a849a52f37"} Nov 03 21:19:10.530056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:10.529677 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa3f2105506f46e9dc7712fd2b575513fbae9c49e55bd51882810a849a52f37" Nov 03 21:19:10.531061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:10.531041 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" event={"ID":"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6","Type":"ContainerStarted","Data":"9a7c824d5d2b2e49bf90422b345194cae5558a339ff7ee2aad9763435877831b"} Nov 03 21:19:10.957670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:10.957601 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:11.178753 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:11.178720 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:13.926111 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:13.926080 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:14.140502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:14.140467 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:14.915448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:14.915408 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:15.036089 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036052 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7"] Nov 03 21:19:15.036649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036341 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerName="extract" Nov 03 21:19:15.036649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036351 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerName="extract" Nov 03 21:19:15.036649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036373 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerName="pull" Nov 03 21:19:15.036649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036378 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerName="pull" Nov 03 21:19:15.036649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036385 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerName="util" Nov 03 21:19:15.036649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036390 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerName="util" Nov 03 21:19:15.036649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.036446 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8467049f-d0a4-4f96-9fd8-78f3c8ead4d9" containerName="extract" Nov 03 21:19:15.039836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.039812 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.092209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.092174 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 21:19:15.141344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.141309 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 21:19:15.141512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.141387 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 21:19:15.141512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.141428 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 21:19:15.141512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.141429 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-tmvnd\"" Nov 03 21:19:15.141512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.141472 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 21:19:15.146291 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.145832 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7"] Nov 03 21:19:15.173272 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.173204 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.173272 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.173268 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jbv\" (UniqueName: \"kubernetes.io/projected/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-kube-api-access-m4jbv\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.173423 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.173390 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.173461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.173433 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.173461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.173452 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.173545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.173528 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.270754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.270728 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:15.274814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.274795 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.274905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.274830 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.274905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.274852 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.274905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.274878 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.275040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.274907 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.275040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.274951 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jbv\" (UniqueName: \"kubernetes.io/projected/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-kube-api-access-m4jbv\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.275298 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.275277 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.277113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.277089 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.277356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.277340 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.277782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.277764 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.277782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.277774 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.333000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.332970 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jbv\" (UniqueName: \"kubernetes.io/projected/5a57cae6-e9d1-4f31-a0d8-e5389329b7a2-kube-api-access-m4jbv\") pod \"opendatahub-operator-controller-manager-5745664b78-q6ls7\" (UID: \"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.357934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.357906 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:15.552951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.552861 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" event={"ID":"62bf39a0-1fce-42c4-82cd-2ef6b897ccb6","Type":"ContainerStarted","Data":"a72fea846b99cf39d5a2c322a224568e1162ad9db4df80eca66ba5302faafadb"} Nov 03 21:19:15.575639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.575596 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:15.673909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.673855 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nzct2" podStartSLOduration=1.60761317 podStartE2EDuration="6.67384047s" totalImagesPullingTime="5.0662273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:15.672983419 +0000 UTC m=+223.283831231" watchObservedRunningTime="2026-11-03 21:19:15.67384047 +0000 UTC m=+223.284688285" Nov 03 21:19:15.786194 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.786162 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:15.849320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:15.849289 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7"] Nov 03 21:19:15.853171 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:15.853144 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a57cae6_e9d1_4f31_a0d8_e5389329b7a2.slice/crio-d9dbcce7d8c5a5846bfebcc2c53967cbb05f6c62b1cf34271c17c66befe4184d WatchSource:0}: Error finding container d9dbcce7d8c5a5846bfebcc2c53967cbb05f6c62b1cf34271c17c66befe4184d: Status 404 returned error can't find the container with id d9dbcce7d8c5a5846bfebcc2c53967cbb05f6c62b1cf34271c17c66befe4184d Nov 03 21:19:16.557710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:16.557675 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" event={"ID":"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2","Type":"ContainerStarted","Data":"d9dbcce7d8c5a5846bfebcc2c53967cbb05f6c62b1cf34271c17c66befe4184d"} Nov 03 21:19:17.061172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:17.061141 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:17.341271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:17.341233 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:17.957494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:17.957440 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:18.234979 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.234902 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:18.382469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.382440 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d"] Nov 03 21:19:18.388308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.388283 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.462911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.462885 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4bq8n\"" Nov 03 21:19:18.498020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.497941 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 21:19:18.498020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.498010 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 21:19:18.504594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.504571 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d"] Nov 03 21:19:18.504921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.504900 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr2zj\" (UniqueName: \"kubernetes.io/projected/bc35cc1e-3a8b-423a-ada5-34d99cc05973-kube-api-access-nr2zj\") pod \"cert-manager-webhook-5dbbbf75cc-qcm8d\" (UID: \"bc35cc1e-3a8b-423a-ada5-34d99cc05973\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.505036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.504982 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc35cc1e-3a8b-423a-ada5-34d99cc05973-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qcm8d\" (UID: \"bc35cc1e-3a8b-423a-ada5-34d99cc05973\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.559990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.559958 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:18.605681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.605645 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc35cc1e-3a8b-423a-ada5-34d99cc05973-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qcm8d\" (UID: \"bc35cc1e-3a8b-423a-ada5-34d99cc05973\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.605852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.605719 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr2zj\" (UniqueName: \"kubernetes.io/projected/bc35cc1e-3a8b-423a-ada5-34d99cc05973-kube-api-access-nr2zj\") pod \"cert-manager-webhook-5dbbbf75cc-qcm8d\" (UID: \"bc35cc1e-3a8b-423a-ada5-34d99cc05973\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.665629 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.665580 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv"] Nov 03 21:19:18.669797 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.669773 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc35cc1e-3a8b-423a-ada5-34d99cc05973-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qcm8d\" (UID: \"bc35cc1e-3a8b-423a-ada5-34d99cc05973\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.670108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.670089 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr2zj\" (UniqueName: \"kubernetes.io/projected/bc35cc1e-3a8b-423a-ada5-34d99cc05973-kube-api-access-nr2zj\") pod \"cert-manager-webhook-5dbbbf75cc-qcm8d\" (UID: \"bc35cc1e-3a8b-423a-ada5-34d99cc05973\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.670588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.670571 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:18.709958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.709929 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:18.725023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.724984 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-h4p99\"" Nov 03 21:19:18.727470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.727411 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv"] Nov 03 21:19:18.807882 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.807850 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2325c2a0-e613-49b7-8093-0b7813636a63-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wv8jv\" (UID: \"2325c2a0-e613-49b7-8093-0b7813636a63\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:18.808023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.807913 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6w7r\" (UniqueName: \"kubernetes.io/projected/2325c2a0-e613-49b7-8093-0b7813636a63-kube-api-access-h6w7r\") pod \"cert-manager-cainjector-5c57bbd4fc-wv8jv\" (UID: \"2325c2a0-e613-49b7-8093-0b7813636a63\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:18.909194 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.909156 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6w7r\" (UniqueName: \"kubernetes.io/projected/2325c2a0-e613-49b7-8093-0b7813636a63-kube-api-access-h6w7r\") pod \"cert-manager-cainjector-5c57bbd4fc-wv8jv\" (UID: \"2325c2a0-e613-49b7-8093-0b7813636a63\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:18.909353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.909249 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2325c2a0-e613-49b7-8093-0b7813636a63-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wv8jv\" (UID: \"2325c2a0-e613-49b7-8093-0b7813636a63\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:18.966191 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.966153 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2325c2a0-e613-49b7-8093-0b7813636a63-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wv8jv\" (UID: \"2325c2a0-e613-49b7-8093-0b7813636a63\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:18.966606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.966517 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6w7r\" (UniqueName: \"kubernetes.io/projected/2325c2a0-e613-49b7-8093-0b7813636a63-kube-api-access-h6w7r\") pod \"cert-manager-cainjector-5c57bbd4fc-wv8jv\" (UID: \"2325c2a0-e613-49b7-8093-0b7813636a63\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:18.999282 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:18.999252 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" Nov 03 21:19:19.094670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:19.094639 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d"] Nov 03 21:19:19.098376 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:19.098337 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc35cc1e_3a8b_423a_ada5_34d99cc05973.slice/crio-eb647521b832137a2781a7af4c8741df44b0088c75f328ef052180c666f79d85 WatchSource:0}: Error finding container eb647521b832137a2781a7af4c8741df44b0088c75f328ef052180c666f79d85: Status 404 returned error can't find the container with id eb647521b832137a2781a7af4c8741df44b0088c75f328ef052180c666f79d85 Nov 03 21:19:19.331687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:19.331598 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:19.371659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:19.371627 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv"] Nov 03 21:19:19.374946 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:19.374913 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2325c2a0_e613_49b7_8093_0b7813636a63.slice/crio-f6a2dc9cc14ccdf569c266c66add448caa622eb9804173aea9d08ec27322ebfa WatchSource:0}: Error finding container f6a2dc9cc14ccdf569c266c66add448caa622eb9804173aea9d08ec27322ebfa: Status 404 returned error can't find the container with id f6a2dc9cc14ccdf569c266c66add448caa622eb9804173aea9d08ec27322ebfa Nov 03 21:19:19.573843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:19.573798 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" event={"ID":"2325c2a0-e613-49b7-8093-0b7813636a63","Type":"ContainerStarted","Data":"f6a2dc9cc14ccdf569c266c66add448caa622eb9804173aea9d08ec27322ebfa"} Nov 03 21:19:19.575148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:19.575123 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" event={"ID":"bc35cc1e-3a8b-423a-ada5-34d99cc05973","Type":"ContainerStarted","Data":"eb647521b832137a2781a7af4c8741df44b0088c75f328ef052180c666f79d85"} Nov 03 21:19:19.867369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:19.852829 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:20.073467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.073438 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g"] Nov 03 21:19:20.078447 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.078427 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.132650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.132575 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:19:20.132650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.132597 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-g2khm\"" Nov 03 21:19:20.132650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.132603 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 21:19:20.134164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.134143 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g"] Nov 03 21:19:20.134261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.134178 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:20.221154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.221123 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtjf\" (UniqueName: \"kubernetes.io/projected/7eb6c775-c6b7-4a4f-844f-b61b429c0a35-kube-api-access-whtjf\") pod \"jobset-operator-674cd59dc5-zgl7g\" (UID: \"7eb6c775-c6b7-4a4f-844f-b61b429c0a35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.221310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.221175 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eb6c775-c6b7-4a4f-844f-b61b429c0a35-tmp\") pod \"jobset-operator-674cd59dc5-zgl7g\" (UID: \"7eb6c775-c6b7-4a4f-844f-b61b429c0a35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.321922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.321888 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eb6c775-c6b7-4a4f-844f-b61b429c0a35-tmp\") pod \"jobset-operator-674cd59dc5-zgl7g\" (UID: \"7eb6c775-c6b7-4a4f-844f-b61b429c0a35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.322124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.321986 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtjf\" (UniqueName: \"kubernetes.io/projected/7eb6c775-c6b7-4a4f-844f-b61b429c0a35-kube-api-access-whtjf\") pod \"jobset-operator-674cd59dc5-zgl7g\" (UID: \"7eb6c775-c6b7-4a4f-844f-b61b429c0a35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.322317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.322297 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eb6c775-c6b7-4a4f-844f-b61b429c0a35-tmp\") pod \"jobset-operator-674cd59dc5-zgl7g\" (UID: \"7eb6c775-c6b7-4a4f-844f-b61b429c0a35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.378463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.378431 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtjf\" (UniqueName: \"kubernetes.io/projected/7eb6c775-c6b7-4a4f-844f-b61b429c0a35-kube-api-access-whtjf\") pod \"jobset-operator-674cd59dc5-zgl7g\" (UID: \"7eb6c775-c6b7-4a4f-844f-b61b429c0a35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.388913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.388865 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" Nov 03 21:19:20.581324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.581288 5931 generic.go:356] "Generic (PLEG): container finished" podID="5a57cae6-e9d1-4f31-a0d8-e5389329b7a2" containerID="521734706b795c217dba04288a0c8ba8811459201a13ee37ef7375b40432b713" exitCode=0 Nov 03 21:19:20.581324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.581325 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" event={"ID":"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2","Type":"ContainerDied","Data":"521734706b795c217dba04288a0c8ba8811459201a13ee37ef7375b40432b713"} Nov 03 21:19:20.770284 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:20.770212 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g"] Nov 03 21:19:20.773413 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:20.773387 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eb6c775_c6b7_4a4f_844f_b61b429c0a35.slice/crio-2b422ebe55d8b40d8aca0e0a489044b93d6f2d46448dd7fb09f9d221ec6e6bd7 WatchSource:0}: Error finding container 2b422ebe55d8b40d8aca0e0a489044b93d6f2d46448dd7fb09f9d221ec6e6bd7: Status 404 returned error can't find the container with id 2b422ebe55d8b40d8aca0e0a489044b93d6f2d46448dd7fb09f9d221ec6e6bd7 Nov 03 21:19:21.589424 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:21.589387 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" event={"ID":"5a57cae6-e9d1-4f31-a0d8-e5389329b7a2","Type":"ContainerStarted","Data":"671bd7bfa483a758a784cc156c44b5a272f2a23c60ec8dc5eb37e4f63c852670"} Nov 03 21:19:21.589870 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:21.589529 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:21.591352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:21.591318 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" event={"ID":"7eb6c775-c6b7-4a4f-844f-b61b429c0a35","Type":"ContainerStarted","Data":"2b422ebe55d8b40d8aca0e0a489044b93d6f2d46448dd7fb09f9d221ec6e6bd7"} Nov 03 21:19:21.842457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:21.842399 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" podStartSLOduration=2.556491511 podStartE2EDuration="7.84238319s" totalImagesPullingTime="4.848356006s" totalInitContainerRuntime="437.535673ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:21.792424255 +0000 UTC m=+229.403272080" watchObservedRunningTime="2026-11-03 21:19:21.84238319 +0000 UTC m=+229.453231068" Nov 03 21:19:24.361275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.361250 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:24.597247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.597214 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:24.608478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.608384 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" event={"ID":"2325c2a0-e613-49b7-8093-0b7813636a63","Type":"ContainerStarted","Data":"f09087d0b5e184edb92eec9eb1efbfd145287d09e00d8584e6d06c62a2891dc2"} Nov 03 21:19:24.610271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.610237 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" event={"ID":"bc35cc1e-3a8b-423a-ada5-34d99cc05973","Type":"ContainerStarted","Data":"6f5aa58734906be1ba5cb7a45b14754d5cb77966af17ca5b9360d59191df94cf"} Nov 03 21:19:24.610398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.610382 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:24.612198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.612175 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" event={"ID":"7eb6c775-c6b7-4a4f-844f-b61b429c0a35","Type":"ContainerStarted","Data":"258dac41798f7e2bd2a19c5f721052ec3edcf5a050e7dd80ab4d3160551f8647"} Nov 03 21:19:24.728743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.728680 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wv8jv" podStartSLOduration=2.259847144 podStartE2EDuration="6.728662595s" totalImagesPullingTime="4.468815451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:24.726534635 +0000 UTC m=+232.337382451" watchObservedRunningTime="2026-11-03 21:19:24.728662595 +0000 UTC m=+232.339510410" Nov 03 21:19:24.842934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.842870 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zgl7g" podStartSLOduration=1.268143215 podStartE2EDuration="4.842851111s" totalImagesPullingTime="3.574707896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:24.841176844 +0000 UTC m=+232.452024659" watchObservedRunningTime="2026-11-03 21:19:24.842851111 +0000 UTC m=+232.453698929" Nov 03 21:19:24.954789 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:24.954694 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" podStartSLOduration=2.059254807 podStartE2EDuration="6.954679398s" totalImagesPullingTime="4.895424591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:24.95343035 +0000 UTC m=+232.564278164" watchObservedRunningTime="2026-11-03 21:19:24.954679398 +0000 UTC m=+232.565527215" Nov 03 21:19:25.602444 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:25.602413 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:25.831407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:25.831378 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:26.999632 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:26.999504 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:28.007742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:28.007704 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:28.367909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:28.367876 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:30.619035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:30.619006 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qcm8d" Nov 03 21:19:32.315459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.315400 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:32.331137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.331110 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dnp2h"] Nov 03 21:19:32.338733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.338711 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:32.389891 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.389864 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dnp2h"] Nov 03 21:19:32.434470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.434440 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcw6x\" (UniqueName: \"kubernetes.io/projected/37f406f3-d2fd-44b6-8cb7-b66aa291772b-kube-api-access-qcw6x\") pod \"cert-manager-597b95cb84-dnp2h\" (UID: \"37f406f3-d2fd-44b6-8cb7-b66aa291772b\") " pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:32.434639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.434481 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37f406f3-d2fd-44b6-8cb7-b66aa291772b-bound-sa-token\") pod \"cert-manager-597b95cb84-dnp2h\" (UID: \"37f406f3-d2fd-44b6-8cb7-b66aa291772b\") " pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:32.437793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.437777 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vlsgd\"" Nov 03 21:19:32.534942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.534917 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcw6x\" (UniqueName: \"kubernetes.io/projected/37f406f3-d2fd-44b6-8cb7-b66aa291772b-kube-api-access-qcw6x\") pod \"cert-manager-597b95cb84-dnp2h\" (UID: \"37f406f3-d2fd-44b6-8cb7-b66aa291772b\") " pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:32.535124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.534967 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37f406f3-d2fd-44b6-8cb7-b66aa291772b-bound-sa-token\") pod \"cert-manager-597b95cb84-dnp2h\" (UID: \"37f406f3-d2fd-44b6-8cb7-b66aa291772b\") " pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:32.594205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.594172 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37f406f3-d2fd-44b6-8cb7-b66aa291772b-bound-sa-token\") pod \"cert-manager-597b95cb84-dnp2h\" (UID: \"37f406f3-d2fd-44b6-8cb7-b66aa291772b\") " pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:32.594451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.594426 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcw6x\" (UniqueName: \"kubernetes.io/projected/37f406f3-d2fd-44b6-8cb7-b66aa291772b-kube-api-access-qcw6x\") pod \"cert-manager-597b95cb84-dnp2h\" (UID: \"37f406f3-d2fd-44b6-8cb7-b66aa291772b\") " pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:32.598768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.598748 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-q6ls7" Nov 03 21:19:32.702512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.702484 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vlsgd\"" Nov 03 21:19:32.709004 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:32.708983 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dnp2h" Nov 03 21:19:33.067756 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:19:33.067676 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:19:33.076190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:33.076164 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dnp2h"] Nov 03 21:19:33.079956 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:33.079932 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37f406f3_d2fd_44b6_8cb7_b66aa291772b.slice/crio-e4207fff789c470d4972c5e0de581b2fad87493693c93e8daeea396fc5c56d47 WatchSource:0}: Error finding container e4207fff789c470d4972c5e0de581b2fad87493693c93e8daeea396fc5c56d47: Status 404 returned error can't find the container with id e4207fff789c470d4972c5e0de581b2fad87493693c93e8daeea396fc5c56d47 Nov 03 21:19:33.651678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:33.651642 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dnp2h" event={"ID":"37f406f3-d2fd-44b6-8cb7-b66aa291772b","Type":"ContainerStarted","Data":"48ca8bd37a433fd5a325c9f148d4462052f2e6074942b442b663fa1afa62511e"} Nov 03 21:19:33.652134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:33.651688 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dnp2h" event={"ID":"37f406f3-d2fd-44b6-8cb7-b66aa291772b","Type":"ContainerStarted","Data":"e4207fff789c470d4972c5e0de581b2fad87493693c93e8daeea396fc5c56d47"} Nov 03 21:19:33.766658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:33.766592 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dnp2h" podStartSLOduration=1.766577233 podStartE2EDuration="1.766577233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:33.765336199 +0000 UTC m=+241.376184015" watchObservedRunningTime="2026-11-03 21:19:33.766577233 +0000 UTC m=+241.377425047" Nov 03 21:19:33.974096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:33.974007 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:35.541895 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:35.541858 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:35.963396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:35.963360 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:36.858214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:36.858182 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:37.968059 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:37.968017 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:38.348018 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:38.347989 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:39.140444 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:39.140409 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:39.408106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:39.408030 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:39.726535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:39.726465 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:39.986372 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:39.986291 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:44.359952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:44.359919 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:44.601329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:44.601300 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:44.838314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:44.838230 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:48.120275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:48.120236 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:48.339078 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:48.339046 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:49.576233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:49.576196 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:49.810749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:49.810725 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:50.606550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:50.606515 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:50.818427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:50.818394 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:51.065385 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:51.065299 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:52.441805 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.441772 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-545b54f677-jpckr"] Nov 03 21:19:52.445908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.445887 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.461215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.461193 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:52.489011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.488984 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8a368e-ca24-45ba-b760-500cab1484f4-tls-certs\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.489140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.489033 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2qt\" (UniqueName: \"kubernetes.io/projected/3b8a368e-ca24-45ba-b760-500cab1484f4-kube-api-access-lf2qt\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.489140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.489127 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b8a368e-ca24-45ba-b760-500cab1484f4-tmp\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.505647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.505604 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545b54f677-jpckr"] Nov 03 21:19:52.549569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.549540 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 21:19:52.549569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.549549 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 21:19:52.553451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.553421 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 21:19:52.553722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.553684 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 21:19:52.553722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.553720 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsllj\"" Nov 03 21:19:52.590239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.590207 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2qt\" (UniqueName: \"kubernetes.io/projected/3b8a368e-ca24-45ba-b760-500cab1484f4-kube-api-access-lf2qt\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.590381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.590254 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b8a368e-ca24-45ba-b760-500cab1484f4-tmp\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.590471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.590454 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8a368e-ca24-45ba-b760-500cab1484f4-tls-certs\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.592576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.592555 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b8a368e-ca24-45ba-b760-500cab1484f4-tmp\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.593173 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.593146 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8a368e-ca24-45ba-b760-500cab1484f4-tls-certs\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.646399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.646375 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2qt\" (UniqueName: \"kubernetes.io/projected/3b8a368e-ca24-45ba-b760-500cab1484f4-kube-api-access-lf2qt\") pod \"kube-auth-proxy-545b54f677-jpckr\" (UID: \"3b8a368e-ca24-45ba-b760-500cab1484f4\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:52.668164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.668130 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:52.758509 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:52.758433 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" Nov 03 21:19:53.051845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:53.051771 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:53.128153 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:53.128123 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545b54f677-jpckr"] Nov 03 21:19:53.133577 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:53.133545 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8a368e_ca24_45ba_b760_500cab1484f4.slice/crio-6c6800edb258e0b6567bebe0df0f3d9eed1ac1da43a96a916e152fa1c89d7134 WatchSource:0}: Error finding container 6c6800edb258e0b6567bebe0df0f3d9eed1ac1da43a96a916e152fa1c89d7134: Status 404 returned error can't find the container with id 6c6800edb258e0b6567bebe0df0f3d9eed1ac1da43a96a916e152fa1c89d7134 Nov 03 21:19:53.282053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:53.282020 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:53.528009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:53.527973 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:53.740130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:53.740088 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" event={"ID":"3b8a368e-ca24-45ba-b760-500cab1484f4","Type":"ContainerStarted","Data":"6c6800edb258e0b6567bebe0df0f3d9eed1ac1da43a96a916e152fa1c89d7134"} Nov 03 21:19:56.115933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:56.115899 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:56.346075 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:56.346045 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:56.755969 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:56.755888 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" event={"ID":"3b8a368e-ca24-45ba-b760-500cab1484f4","Type":"ContainerStarted","Data":"891ed773251834384792af1a6f6196bde7feefabb8bf36142fb816985ef6415a"} Nov 03 21:19:56.888480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:56.888411 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-545b54f677-jpckr" podStartSLOduration=1.5272149769999999 podStartE2EDuration="4.888394107s" totalImagesPullingTime="3.36117913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:56.887043165 +0000 UTC m=+264.497890978" watchObservedRunningTime="2026-11-03 21:19:56.888394107 +0000 UTC m=+264.499241922" Nov 03 21:19:57.031205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:57.031123 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:59.132424 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.132392 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mjjzm"] Nov 03 21:19:59.132830 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.132392 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:59.134650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.134636 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.187240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.187207 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 21:19:59.187394 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.187310 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 21:19:59.236930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.236902 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 21:19:59.237107 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.237087 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvf9h\"" Nov 03 21:19:59.238316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.238296 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mjjzm"] Nov 03 21:19:59.255954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.255918 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9grs\" (UniqueName: \"kubernetes.io/projected/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-kube-api-access-f9grs\") pod \"limitador-limitador-74dfcfb894-mjjzm\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.255954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.255959 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-config-file\") pod \"limitador-limitador-74dfcfb894-mjjzm\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.357179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.357135 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9grs\" (UniqueName: \"kubernetes.io/projected/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-kube-api-access-f9grs\") pod \"limitador-limitador-74dfcfb894-mjjzm\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.357337 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.357196 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-config-file\") pod \"limitador-limitador-74dfcfb894-mjjzm\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.357960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.357937 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-config-file\") pod \"limitador-limitador-74dfcfb894-mjjzm\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.377059 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.377032 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mjjzm"] Nov 03 21:19:59.415294 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.415227 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9grs\" (UniqueName: \"kubernetes.io/projected/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-kube-api-access-f9grs\") pod \"limitador-limitador-74dfcfb894-mjjzm\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.446106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.446074 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:19:59.488962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.488928 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:59.722318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.722237 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:59.816980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:19:59.816951 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mjjzm"] Nov 03 21:19:59.821155 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:19:59.821127 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc62955_fa1a_405d_9dbd_d05fbfb247c2.slice/crio-f479e214efdf9c6c2d569e91decffa804d97aa87b7c8e00dad651a880295fa80 WatchSource:0}: Error finding container f479e214efdf9c6c2d569e91decffa804d97aa87b7c8e00dad651a880295fa80: Status 404 returned error can't find the container with id f479e214efdf9c6c2d569e91decffa804d97aa87b7c8e00dad651a880295fa80 Nov 03 21:20:00.578914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:00.578882 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:00.779086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:00.779056 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" event={"ID":"bdc62955-fa1a-405d-9dbd-d05fbfb247c2","Type":"ContainerStarted","Data":"f479e214efdf9c6c2d569e91decffa804d97aa87b7c8e00dad651a880295fa80"} Nov 03 21:20:00.898827 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:00.898787 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:01.210935 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:01.210824 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:03.066965 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:20:03.066931 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:20:03.314151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.314119 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:03.374373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.374337 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jctxd"] Nov 03 21:20:03.377213 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.377190 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" Nov 03 21:20:03.430768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.430742 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jctxd"] Nov 03 21:20:03.478327 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.478293 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t72bw\"" Nov 03 21:20:03.491572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.491544 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpgh\" (UniqueName: \"kubernetes.io/projected/1b541dba-a32f-48f3-8044-ba386a39cd2c-kube-api-access-bgpgh\") pod \"authorino-56c6c9ddd5-jctxd\" (UID: \"1b541dba-a32f-48f3-8044-ba386a39cd2c\") " pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" Nov 03 21:20:03.529371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.529345 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:03.592709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.592682 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpgh\" (UniqueName: \"kubernetes.io/projected/1b541dba-a32f-48f3-8044-ba386a39cd2c-kube-api-access-bgpgh\") pod \"authorino-56c6c9ddd5-jctxd\" (UID: \"1b541dba-a32f-48f3-8044-ba386a39cd2c\") " pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" Nov 03 21:20:03.648902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.648838 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpgh\" (UniqueName: \"kubernetes.io/projected/1b541dba-a32f-48f3-8044-ba386a39cd2c-kube-api-access-bgpgh\") pod \"authorino-56c6c9ddd5-jctxd\" (UID: \"1b541dba-a32f-48f3-8044-ba386a39cd2c\") " pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" Nov 03 21:20:03.687726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.687696 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" Nov 03 21:20:03.734920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.734728 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:03.948908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:03.948841 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:04.052721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.052656 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jctxd"] Nov 03 21:20:04.056598 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:20:04.056567 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b541dba_a32f_48f3_8044_ba386a39cd2c.slice/crio-d0830f51faeaa7c49d1bea16457547b72ad5d191debfe0f4aea5fb0611c733bd WatchSource:0}: Error finding container d0830f51faeaa7c49d1bea16457547b72ad5d191debfe0f4aea5fb0611c733bd: Status 404 returned error can't find the container with id d0830f51faeaa7c49d1bea16457547b72ad5d191debfe0f4aea5fb0611c733bd Nov 03 21:20:04.534670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.534632 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:04.736362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.736334 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv"] Nov 03 21:20:04.739299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.739279 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.757863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.757830 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:04.800704 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.800382 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" event={"ID":"1b541dba-a32f-48f3-8044-ba386a39cd2c","Type":"ContainerStarted","Data":"d0830f51faeaa7c49d1bea16457547b72ad5d191debfe0f4aea5fb0611c733bd"} Nov 03 21:20:04.803859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.803827 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgcs\" (UniqueName: \"kubernetes.io/projected/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-kube-api-access-gdgcs\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.803976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.803871 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-manager-config\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.803976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.803908 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-metrics-certs\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.804059 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.803987 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-cert\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.844134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.844091 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 21:20:04.844331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.844222 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 21:20:04.844331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.844257 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 21:20:04.844331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.844287 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-fcdgc\"" Nov 03 21:20:04.850632 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.846129 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv"] Nov 03 21:20:04.905133 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.905103 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-cert\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.905276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.905179 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgcs\" (UniqueName: \"kubernetes.io/projected/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-kube-api-access-gdgcs\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.905276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.905208 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-manager-config\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.905276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.905225 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-metrics-certs\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.905927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.905905 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-manager-config\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.907682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.907666 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-metrics-certs\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.907769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.907750 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-cert\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:04.961734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:04.961705 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgcs\" (UniqueName: \"kubernetes.io/projected/311a46f3-1cb9-4e16-8a3f-5813e2b4a51f-kube-api-access-gdgcs\") pod \"jobset-controller-manager-6f78675b48-sd8dv\" (UID: \"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f\") " pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:05.050926 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:05.050851 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:05.418403 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:05.418368 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv"] Nov 03 21:20:05.421535 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:20:05.421511 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311a46f3_1cb9_4e16_8a3f_5813e2b4a51f.slice/crio-d00bebf5cb46d07a10e593ebb611a41e5144ae2bf1c98bc673402c879be9b847 WatchSource:0}: Error finding container d00bebf5cb46d07a10e593ebb611a41e5144ae2bf1c98bc673402c879be9b847: Status 404 returned error can't find the container with id d00bebf5cb46d07a10e593ebb611a41e5144ae2bf1c98bc673402c879be9b847 Nov 03 21:20:05.806149 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:05.806070 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" event={"ID":"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f","Type":"ContainerStarted","Data":"d00bebf5cb46d07a10e593ebb611a41e5144ae2bf1c98bc673402c879be9b847"} Nov 03 21:20:07.932021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:07.931985 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:08.133473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:08.133402 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:09.827746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:09.827709 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" event={"ID":"1b541dba-a32f-48f3-8044-ba386a39cd2c","Type":"ContainerStarted","Data":"68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec"} Nov 03 21:20:09.940518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:09.940450 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" podStartSLOduration=2.131285583 podStartE2EDuration="6.940431756s" totalImagesPullingTime="4.809146173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:09.939541378 +0000 UTC m=+277.550389192" watchObservedRunningTime="2026-11-03 21:20:09.940431756 +0000 UTC m=+277.551279569" Nov 03 21:20:10.327643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:10.326668 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:10.538248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:10.538213 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:11.839066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:11.839031 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" event={"ID":"311a46f3-1cb9-4e16-8a3f-5813e2b4a51f","Type":"ContainerStarted","Data":"455055fc3be8be376d824fb45a88c15110819aa0d6da00fc74201b9cd9a0df65"} Nov 03 21:20:11.839516 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:11.839122 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:11.954101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:11.954001 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" podStartSLOduration=1.721695976 podStartE2EDuration="7.95398657s" totalImagesPullingTime="6.232290594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:11.953489296 +0000 UTC m=+279.564337109" watchObservedRunningTime="2026-11-03 21:20:11.95398657 +0000 UTC m=+279.564834384" Nov 03 21:20:13.850798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:13.850764 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" event={"ID":"bdc62955-fa1a-405d-9dbd-d05fbfb247c2","Type":"ContainerStarted","Data":"3078c23292cbd0250d5d3faf7dd30c0f9197868b0f05832241c7c3c2d2d9fa70"} Nov 03 21:20:13.851282 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:13.850891 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:20:13.964061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:13.964010 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" podStartSLOduration=1.328318635 podStartE2EDuration="14.963995085s" totalImagesPullingTime="13.63567645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:13.96212316 +0000 UTC m=+281.572970974" watchObservedRunningTime="2026-11-03 21:20:13.963995085 +0000 UTC m=+281.574842899" Nov 03 21:20:16.079597 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:16.079560 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:16.286594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:16.286556 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:20.362159 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:20.362126 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mjjzm"] Nov 03 21:20:20.362670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:20.362301 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" podUID="bdc62955-fa1a-405d-9dbd-d05fbfb247c2" containerName="limitador" containerID="cri-o://3078c23292cbd0250d5d3faf7dd30c0f9197868b0f05832241c7c3c2d2d9fa70" gracePeriod=30 Nov 03 21:20:20.363030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:20.362974 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:20:20.887006 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:20.886970 5931 generic.go:356] "Generic (PLEG): container finished" podID="bdc62955-fa1a-405d-9dbd-d05fbfb247c2" containerID="3078c23292cbd0250d5d3faf7dd30c0f9197868b0f05832241c7c3c2d2d9fa70" exitCode=0 Nov 03 21:20:20.887150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:20.887068 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" event={"ID":"bdc62955-fa1a-405d-9dbd-d05fbfb247c2","Type":"ContainerDied","Data":"3078c23292cbd0250d5d3faf7dd30c0f9197868b0f05832241c7c3c2d2d9fa70"} Nov 03 21:20:20.904817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:20.904795 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:20:21.055729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.055651 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-config-file\" (UniqueName: \"kubernetes.io/configmap/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-config-file\") pod \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " Nov 03 21:20:21.055892 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.055812 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-kube-api-access-f9grs\" (UniqueName: \"kubernetes.io/projected/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-kube-api-access-f9grs\") pod \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\" (UID: \"bdc62955-fa1a-405d-9dbd-d05fbfb247c2\") " Nov 03 21:20:21.056067 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.056041 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-config-file" pod "bdc62955-fa1a-405d-9dbd-d05fbfb247c2" (UID: "bdc62955-fa1a-405d-9dbd-d05fbfb247c2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:21.058066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.058039 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-kube-api-access-f9grs" pod "bdc62955-fa1a-405d-9dbd-d05fbfb247c2" (UID: "bdc62955-fa1a-405d-9dbd-d05fbfb247c2"). InnerVolumeSpecName "kube-api-access-f9grs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:21.157161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.157130 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9grs\" (UniqueName: \"kubernetes.io/projected/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-kube-api-access-f9grs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:21.157161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.157156 5931 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdc62955-fa1a-405d-9dbd-d05fbfb247c2-config-file\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:21.892565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.892531 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" event={"ID":"bdc62955-fa1a-405d-9dbd-d05fbfb247c2","Type":"ContainerDied","Data":"f479e214efdf9c6c2d569e91decffa804d97aa87b7c8e00dad651a880295fa80"} Nov 03 21:20:21.892565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.892552 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mjjzm" Nov 03 21:20:21.893061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:21.892587 5931 scope.go:122] "RemoveContainer" containerID="3078c23292cbd0250d5d3faf7dd30c0f9197868b0f05832241c7c3c2d2d9fa70" Nov 03 21:20:22.054387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:22.054349 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mjjzm"] Nov 03 21:20:22.108129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:22.105547 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mjjzm"] Nov 03 21:20:22.504909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:22.504869 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc62955-fa1a-405d-9dbd-d05fbfb247c2" path="/var/lib/kubelet/pods/bdc62955-fa1a-405d-9dbd-d05fbfb247c2/volumes" Nov 03 21:20:22.849745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:22.849713 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6f78675b48-sd8dv" Nov 03 21:20:28.494337 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:28.494303 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:28.712749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:28.712716 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:29.259206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:29.259169 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:32.542409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:32.542367 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:20:32.542841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:32.542568 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:20:33.067425 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:20:33.067386 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:20:35.115587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:35.115538 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:35.326063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:35.326026 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:37.176196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:37.176144 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:51.025157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:51.025107 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:51.266720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:51.266682 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:51.487802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:51.487765 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:58.615602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:58.615571 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:58.833321 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:20:58.833289 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:01.770538 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:01.770505 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:01.990738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:01.990699 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:03.066876 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:21:03.066839 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:21:25.891330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:25.891298 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.113587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:26.113556 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.361038 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:26.361006 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.631884 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:26.631800 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.902445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:26.902366 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:27.113397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:27.113368 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:28.155563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:28.155536 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:31.891314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:31.891271 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:32.112639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:32.112584 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:33.067319 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:21:33.067285 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:21:33.280760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:33.280726 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:33.491459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:33.491428 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:33.689942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:33.689906 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:33.898306 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:33.898278 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:34.131389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.131355 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:34.339818 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.339790 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:34.548384 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.548351 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:34.901821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.901787 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4c794755-rkc56"] Nov 03 21:21:34.902123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.902111 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bdc62955-fa1a-405d-9dbd-d05fbfb247c2" containerName="limitador" Nov 03 21:21:34.902174 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.902125 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc62955-fa1a-405d-9dbd-d05fbfb247c2" containerName="limitador" Nov 03 21:21:34.902210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.902204 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc62955-fa1a-405d-9dbd-d05fbfb247c2" containerName="limitador" Nov 03 21:21:34.904034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.904018 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:34.956053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:34.956020 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:21:35.000546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.000518 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-metrics-certs\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.000730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.000555 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1653ff-1df8-4dba-ac05-5979bf6a267a-openshift-service-ca\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.000730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.000707 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-maas-api-tls\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.000827 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.000742 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltts\" (UniqueName: \"kubernetes.io/projected/3b1653ff-1df8-4dba-ac05-5979bf6a267a-kube-api-access-dltts\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.005396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.005379 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:21:35.005396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.005390 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nbmd6\"" Nov 03 21:21:35.005578 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.005429 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 21:21:35.005578 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.005562 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:21:35.007245 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.007224 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4c794755-rkc56"] Nov 03 21:21:35.101878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.101844 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltts\" (UniqueName: \"kubernetes.io/projected/3b1653ff-1df8-4dba-ac05-5979bf6a267a-kube-api-access-dltts\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.102058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.101937 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-metrics-certs\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.102058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.101970 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1653ff-1df8-4dba-ac05-5979bf6a267a-openshift-service-ca\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.102058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.102003 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-maas-api-tls\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.102604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.102582 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1653ff-1df8-4dba-ac05-5979bf6a267a-openshift-service-ca\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.104423 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.104404 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-maas-api-tls\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.104693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.104676 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-metrics-certs\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.158365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.158313 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltts\" (UniqueName: \"kubernetes.io/projected/3b1653ff-1df8-4dba-ac05-5979bf6a267a-kube-api-access-dltts\") pod \"maas-api-7c4c794755-rkc56\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.214941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.214915 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:35.587293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:35.587266 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4c794755-rkc56"] Nov 03 21:21:35.589910 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:21:35.589884 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1653ff_1df8_4dba_ac05_5979bf6a267a.slice/crio-1a8ed53e3d822da6d3ac5f80a5339987c52ea744d8281c413049998efc8ba390 WatchSource:0}: Error finding container 1a8ed53e3d822da6d3ac5f80a5339987c52ea744d8281c413049998efc8ba390: Status 404 returned error can't find the container with id 1a8ed53e3d822da6d3ac5f80a5339987c52ea744d8281c413049998efc8ba390 Nov 03 21:21:36.259063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:36.259029 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" event={"ID":"3b1653ff-1df8-4dba-ac05-5979bf6a267a","Type":"ContainerStarted","Data":"1a8ed53e3d822da6d3ac5f80a5339987c52ea744d8281c413049998efc8ba390"} Nov 03 21:21:37.513373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:37.513337 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:42.296103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:42.296015 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" event={"ID":"3b1653ff-1df8-4dba-ac05-5979bf6a267a","Type":"ContainerStarted","Data":"9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092"} Nov 03 21:21:42.296456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:42.296130 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:42.412140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:42.412088 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" podStartSLOduration=2.154641826 podStartE2EDuration="8.412072187s" totalImagesPullingTime="6.257430361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:42.409935322 +0000 UTC m=+370.020783164" watchObservedRunningTime="2026-11-03 21:21:42.412072187 +0000 UTC m=+370.022920002" Nov 03 21:21:48.305836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:48.305807 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:48.937479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:48.937450 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:49.151327 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.151295 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:49.160932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.160905 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hbs6z"] Nov 03 21:21:49.164310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.164292 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" Nov 03 21:21:49.220756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.220697 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hbs6z"] Nov 03 21:21:49.231236 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.231214 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9xb\" (UniqueName: \"kubernetes.io/projected/7922c74c-746b-4a47-b379-5caeef34d777-kube-api-access-qf9xb\") pod \"authorino-6f6944bcdd-hbs6z\" (UID: \"7922c74c-746b-4a47-b379-5caeef34d777\") " pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" Nov 03 21:21:49.331839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.331808 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9xb\" (UniqueName: \"kubernetes.io/projected/7922c74c-746b-4a47-b379-5caeef34d777-kube-api-access-qf9xb\") pod \"authorino-6f6944bcdd-hbs6z\" (UID: \"7922c74c-746b-4a47-b379-5caeef34d777\") " pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" Nov 03 21:21:49.387731 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.387698 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9xb\" (UniqueName: \"kubernetes.io/projected/7922c74c-746b-4a47-b379-5caeef34d777-kube-api-access-qf9xb\") pod \"authorino-6f6944bcdd-hbs6z\" (UID: \"7922c74c-746b-4a47-b379-5caeef34d777\") " pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" Nov 03 21:21:49.475986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.475913 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" Nov 03 21:21:49.543828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.543753 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hbs6z"] Nov 03 21:21:49.847213 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:49.847183 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hbs6z"] Nov 03 21:21:49.850268 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:21:49.850233 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7922c74c_746b_4a47_b379_5caeef34d777.slice/crio-d271895b5dd40ef32ac67a10f7ea253f3a38579e3bd0edc96e72820edb087eff WatchSource:0}: Error finding container d271895b5dd40ef32ac67a10f7ea253f3a38579e3bd0edc96e72820edb087eff: Status 404 returned error can't find the container with id d271895b5dd40ef32ac67a10f7ea253f3a38579e3bd0edc96e72820edb087eff Nov 03 21:21:50.027468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.027438 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5867475f65-zr2tb"] Nov 03 21:21:50.031770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.031750 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.086409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.086380 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5867475f65-zr2tb"] Nov 03 21:21:50.134553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.134484 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 21:21:50.137895 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.137873 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4b582e-b35d-47c2-b017-097ede55a4ff-tls-cert\") pod \"authorino-5867475f65-zr2tb\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.137975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.137919 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnnc\" (UniqueName: \"kubernetes.io/projected/1a4b582e-b35d-47c2-b017-097ede55a4ff-kube-api-access-zfnnc\") pod \"authorino-5867475f65-zr2tb\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.238786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.238750 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4b582e-b35d-47c2-b017-097ede55a4ff-tls-cert\") pod \"authorino-5867475f65-zr2tb\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.238980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.238805 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnnc\" (UniqueName: \"kubernetes.io/projected/1a4b582e-b35d-47c2-b017-097ede55a4ff-kube-api-access-zfnnc\") pod \"authorino-5867475f65-zr2tb\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.241215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.241194 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4b582e-b35d-47c2-b017-097ede55a4ff-tls-cert\") pod \"authorino-5867475f65-zr2tb\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.294881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.294853 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnnc\" (UniqueName: \"kubernetes.io/projected/1a4b582e-b35d-47c2-b017-097ede55a4ff-kube-api-access-zfnnc\") pod \"authorino-5867475f65-zr2tb\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.332698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.332669 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" event={"ID":"7922c74c-746b-4a47-b379-5caeef34d777","Type":"ContainerStarted","Data":"d271895b5dd40ef32ac67a10f7ea253f3a38579e3bd0edc96e72820edb087eff"} Nov 03 21:21:50.342921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.342896 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:21:50.710288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:50.709933 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5867475f65-zr2tb"] Nov 03 21:21:50.712673 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:21:50.712640 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a4b582e_b35d_47c2_b017_097ede55a4ff.slice/crio-6f478c684d0991fc6b4a8b6112049e5701d55686d583c9be4b407f41cc88fc4a WatchSource:0}: Error finding container 6f478c684d0991fc6b4a8b6112049e5701d55686d583c9be4b407f41cc88fc4a: Status 404 returned error can't find the container with id 6f478c684d0991fc6b4a8b6112049e5701d55686d583c9be4b407f41cc88fc4a Nov 03 21:21:51.338910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:51.338870 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5867475f65-zr2tb" event={"ID":"1a4b582e-b35d-47c2-b017-097ede55a4ff","Type":"ContainerStarted","Data":"6f478c684d0991fc6b4a8b6112049e5701d55686d583c9be4b407f41cc88fc4a"} Nov 03 21:21:52.348400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.348362 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5867475f65-zr2tb" event={"ID":"1a4b582e-b35d-47c2-b017-097ede55a4ff","Type":"ContainerStarted","Data":"be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3"} Nov 03 21:21:52.461375 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.461329 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5867475f65-zr2tb" podStartSLOduration=2.2702934470000002 podStartE2EDuration="3.461314669s" totalImagesPullingTime="1.191021222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:52.460145398 +0000 UTC m=+380.070993219" watchObservedRunningTime="2026-11-03 21:21:52.461314669 +0000 UTC m=+380.072162530" Nov 03 21:21:52.485328 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.485293 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jctxd"] Nov 03 21:21:52.485495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.485476 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" podUID="1b541dba-a32f-48f3-8044-ba386a39cd2c" containerName="authorino" containerID="cri-o://68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec" gracePeriod=30 Nov 03 21:21:52.737541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.737518 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" Nov 03 21:21:52.863059 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.862974 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b541dba-a32f-48f3-8044-ba386a39cd2c-kube-api-access-bgpgh\" (UniqueName: \"kubernetes.io/projected/1b541dba-a32f-48f3-8044-ba386a39cd2c-kube-api-access-bgpgh\") pod \"1b541dba-a32f-48f3-8044-ba386a39cd2c\" (UID: \"1b541dba-a32f-48f3-8044-ba386a39cd2c\") " Nov 03 21:21:52.865150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.865123 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b541dba-a32f-48f3-8044-ba386a39cd2c-kube-api-access-bgpgh" pod "1b541dba-a32f-48f3-8044-ba386a39cd2c" (UID: "1b541dba-a32f-48f3-8044-ba386a39cd2c"). InnerVolumeSpecName "kube-api-access-bgpgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:52.964289 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:52.964250 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgpgh\" (UniqueName: \"kubernetes.io/projected/1b541dba-a32f-48f3-8044-ba386a39cd2c-kube-api-access-bgpgh\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:53.354450 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.354419 5931 generic.go:356] "Generic (PLEG): container finished" podID="1b541dba-a32f-48f3-8044-ba386a39cd2c" containerID="68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec" exitCode=0 Nov 03 21:21:53.354922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.354480 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" Nov 03 21:21:53.354922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.354510 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" event={"ID":"1b541dba-a32f-48f3-8044-ba386a39cd2c","Type":"ContainerDied","Data":"68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec"} Nov 03 21:21:53.354922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.354560 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jctxd" event={"ID":"1b541dba-a32f-48f3-8044-ba386a39cd2c","Type":"ContainerDied","Data":"d0830f51faeaa7c49d1bea16457547b72ad5d191debfe0f4aea5fb0611c733bd"} Nov 03 21:21:53.354922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.354584 5931 scope.go:122] "RemoveContainer" containerID="68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec" Nov 03 21:21:53.363322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.363307 5931 scope.go:122] "RemoveContainer" containerID="68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec" Nov 03 21:21:53.363588 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:21:53.363566 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec\": container with ID starting with 68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec not found: ID does not exist" containerID="68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec" Nov 03 21:21:53.363681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.363595 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec"} err="failed to get container status \"68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec\": rpc error: code = NotFound desc = could not find container \"68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec\": container with ID starting with 68e8b507a86973207c20b060daf470cba772f39c5e0a41d25a15f9560611c5ec not found: ID does not exist" Nov 03 21:21:53.515274 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.515238 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jctxd"] Nov 03 21:21:53.517027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:53.517005 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jctxd"] Nov 03 21:21:54.503629 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:54.503591 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b541dba-a32f-48f3-8044-ba386a39cd2c" path="/var/lib/kubelet/pods/1b541dba-a32f-48f3-8044-ba386a39cd2c/volumes" Nov 03 21:21:58.383673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.383607 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" event={"ID":"7922c74c-746b-4a47-b379-5caeef34d777","Type":"ContainerStarted","Data":"3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f"} Nov 03 21:21:58.384148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.383709 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" podUID="7922c74c-746b-4a47-b379-5caeef34d777" containerName="authorino" containerID="cri-o://3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f" gracePeriod=30 Nov 03 21:21:58.494939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.494520 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" podStartSLOduration=1.6444655259999998 podStartE2EDuration="9.494501906s" totalImagesPullingTime="7.85003638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:58.493898484 +0000 UTC m=+386.104746301" watchObservedRunningTime="2026-11-03 21:21:58.494501906 +0000 UTC m=+386.105349721" Nov 03 21:21:58.631601 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.631575 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" Nov 03 21:21:58.717583 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.717510 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7922c74c-746b-4a47-b379-5caeef34d777-kube-api-access-qf9xb\" (UniqueName: \"kubernetes.io/projected/7922c74c-746b-4a47-b379-5caeef34d777-kube-api-access-qf9xb\") pod \"7922c74c-746b-4a47-b379-5caeef34d777\" (UID: \"7922c74c-746b-4a47-b379-5caeef34d777\") " Nov 03 21:21:58.719691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.719667 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7922c74c-746b-4a47-b379-5caeef34d777-kube-api-access-qf9xb" pod "7922c74c-746b-4a47-b379-5caeef34d777" (UID: "7922c74c-746b-4a47-b379-5caeef34d777"). InnerVolumeSpecName "kube-api-access-qf9xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:58.818764 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.818729 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf9xb\" (UniqueName: \"kubernetes.io/projected/7922c74c-746b-4a47-b379-5caeef34d777-kube-api-access-qf9xb\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:58.832830 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.832801 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4c794755-rkc56"] Nov 03 21:21:58.833019 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:58.832994 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" podUID="3b1653ff-1df8-4dba-ac05-5979bf6a267a" containerName="maas-api" containerID="cri-o://9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092" gracePeriod=30 Nov 03 21:21:59.064931 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.064907 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:59.222687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.222656 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-metrics-certs\") pod \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " Nov 03 21:21:59.222852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.222718 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1653ff-1df8-4dba-ac05-5979bf6a267a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1653ff-1df8-4dba-ac05-5979bf6a267a-openshift-service-ca\") pod \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " Nov 03 21:21:59.222852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.222749 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-maas-api-tls\") pod \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " Nov 03 21:21:59.222985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.222851 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1653ff-1df8-4dba-ac05-5979bf6a267a-kube-api-access-dltts\" (UniqueName: \"kubernetes.io/projected/3b1653ff-1df8-4dba-ac05-5979bf6a267a-kube-api-access-dltts\") pod \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\" (UID: \"3b1653ff-1df8-4dba-ac05-5979bf6a267a\") " Nov 03 21:21:59.223120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.223098 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1653ff-1df8-4dba-ac05-5979bf6a267a-openshift-service-ca" pod "3b1653ff-1df8-4dba-ac05-5979bf6a267a" (UID: "3b1653ff-1df8-4dba-ac05-5979bf6a267a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:59.225023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.225004 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-maas-api-tls" pod "3b1653ff-1df8-4dba-ac05-5979bf6a267a" (UID: "3b1653ff-1df8-4dba-ac05-5979bf6a267a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:59.225094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.225077 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1653ff-1df8-4dba-ac05-5979bf6a267a-kube-api-access-dltts" pod "3b1653ff-1df8-4dba-ac05-5979bf6a267a" (UID: "3b1653ff-1df8-4dba-ac05-5979bf6a267a"). InnerVolumeSpecName "kube-api-access-dltts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:59.225154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.225130 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-metrics-certs" pod "3b1653ff-1df8-4dba-ac05-5979bf6a267a" (UID: "3b1653ff-1df8-4dba-ac05-5979bf6a267a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:59.324258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.324177 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:59.324258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.324206 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1653ff-1df8-4dba-ac05-5979bf6a267a-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:59.324258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.324218 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1653ff-1df8-4dba-ac05-5979bf6a267a-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:59.324258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.324231 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dltts\" (UniqueName: \"kubernetes.io/projected/3b1653ff-1df8-4dba-ac05-5979bf6a267a-kube-api-access-dltts\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:59.394784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.394743 5931 generic.go:356] "Generic (PLEG): container finished" podID="3b1653ff-1df8-4dba-ac05-5979bf6a267a" containerID="9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092" exitCode=0 Nov 03 21:21:59.395227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.394784 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" event={"ID":"3b1653ff-1df8-4dba-ac05-5979bf6a267a","Type":"ContainerDied","Data":"9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092"} Nov 03 21:21:59.395227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.394810 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" Nov 03 21:21:59.395227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.394874 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c4c794755-rkc56" event={"ID":"3b1653ff-1df8-4dba-ac05-5979bf6a267a","Type":"ContainerDied","Data":"1a8ed53e3d822da6d3ac5f80a5339987c52ea744d8281c413049998efc8ba390"} Nov 03 21:21:59.395227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.394906 5931 scope.go:122] "RemoveContainer" containerID="9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092" Nov 03 21:21:59.396197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.396172 5931 generic.go:356] "Generic (PLEG): container finished" podID="7922c74c-746b-4a47-b379-5caeef34d777" containerID="3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f" exitCode=0 Nov 03 21:21:59.396316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.396229 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" Nov 03 21:21:59.396316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.396244 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" event={"ID":"7922c74c-746b-4a47-b379-5caeef34d777","Type":"ContainerDied","Data":"3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f"} Nov 03 21:21:59.396316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.396273 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hbs6z" event={"ID":"7922c74c-746b-4a47-b379-5caeef34d777","Type":"ContainerDied","Data":"d271895b5dd40ef32ac67a10f7ea253f3a38579e3bd0edc96e72820edb087eff"} Nov 03 21:21:59.404659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.404638 5931 scope.go:122] "RemoveContainer" containerID="9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092" Nov 03 21:21:59.404915 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:21:59.404894 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092\": container with ID starting with 9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092 not found: ID does not exist" containerID="9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092" Nov 03 21:21:59.404998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.404927 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092"} err="failed to get container status \"9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092\": rpc error: code = NotFound desc = could not find container \"9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092\": container with ID starting with 9dec769cb003034310b9f931961eb7d1ea27be185678a9986b94bb5a602a9092 not found: ID does not exist" Nov 03 21:21:59.404998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.404948 5931 scope.go:122] "RemoveContainer" containerID="3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f" Nov 03 21:21:59.412599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.412573 5931 scope.go:122] "RemoveContainer" containerID="3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f" Nov 03 21:21:59.412897 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:21:59.412879 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f\": container with ID starting with 3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f not found: ID does not exist" containerID="3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f" Nov 03 21:21:59.412957 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.412903 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f"} err="failed to get container status \"3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f\": rpc error: code = NotFound desc = could not find container \"3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f\": container with ID starting with 3de5e3f3d713eb580b341ffe6f0fb82bf2e6ab69a1f6f8cb936f10b51f41ab6f not found: ID does not exist" Nov 03 21:21:59.555866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.555836 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4c794755-rkc56"] Nov 03 21:21:59.605098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.605075 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4c794755-rkc56"] Nov 03 21:21:59.664767 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.664741 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hbs6z"] Nov 03 21:21:59.668394 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:21:59.668372 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hbs6z"] Nov 03 21:22:00.503464 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:22:00.503428 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1653ff-1df8-4dba-ac05-5979bf6a267a" path="/var/lib/kubelet/pods/3b1653ff-1df8-4dba-ac05-5979bf6a267a/volumes" Nov 03 21:22:00.503864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:22:00.503764 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7922c74c-746b-4a47-b379-5caeef34d777" path="/var/lib/kubelet/pods/7922c74c-746b-4a47-b379-5caeef34d777/volumes" Nov 03 21:22:01.618151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:22:01.618118 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:03.066316 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:22:03.066282 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:22:26.053811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:22:26.053776 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:33.067490 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:22:33.067452 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:22:37.863483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:22:37.863442 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:48.204548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:22:48.204513 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:48.461415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:22:48.461338 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:03.067355 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:23:03.067313 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:23:18.331346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:18.331273 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:18.590974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:18.590949 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:25.736374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:25.736337 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:25.980765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:25.980728 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:26.029653 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.029559 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42bpn"] Nov 03 21:23:26.029981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.029963 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7922c74c-746b-4a47-b379-5caeef34d777" containerName="authorino" Nov 03 21:23:26.030065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.029983 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7922c74c-746b-4a47-b379-5caeef34d777" containerName="authorino" Nov 03 21:23:26.030065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.030006 5931 cpu_manager.go:414] "containerMap: removing container" podUID="3b1653ff-1df8-4dba-ac05-5979bf6a267a" containerName="maas-api" Nov 03 21:23:26.030065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.030013 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1653ff-1df8-4dba-ac05-5979bf6a267a" containerName="maas-api" Nov 03 21:23:26.030065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.030023 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1b541dba-a32f-48f3-8044-ba386a39cd2c" containerName="authorino" Nov 03 21:23:26.030065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.030030 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b541dba-a32f-48f3-8044-ba386a39cd2c" containerName="authorino" Nov 03 21:23:26.030299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.030132 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1653ff-1df8-4dba-ac05-5979bf6a267a" containerName="maas-api" Nov 03 21:23:26.030299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.030145 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b541dba-a32f-48f3-8044-ba386a39cd2c" containerName="authorino" Nov 03 21:23:26.030299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.030160 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7922c74c-746b-4a47-b379-5caeef34d777" containerName="authorino" Nov 03 21:23:26.033418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.033399 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.087215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.087192 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 21:23:26.133340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.133307 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8c4e07-e240-416d-a7ca-ddf5c91b9d54-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-42bpn\" (UID: \"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.133340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.133340 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pllmm\" (UniqueName: \"kubernetes.io/projected/8c8c4e07-e240-416d-a7ca-ddf5c91b9d54-kube-api-access-pllmm\") pod \"payload-pre-processing-7b98bb678c-42bpn\" (UID: \"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.136552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.136534 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z9bfl\"" Nov 03 21:23:26.138092 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.138072 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42bpn"] Nov 03 21:23:26.195711 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.195678 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-chklw"] Nov 03 21:23:26.199393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.199372 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.233823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.233794 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmwpw\" (UniqueName: \"kubernetes.io/projected/20bf40f5-8b81-478e-9e5b-c0fa554aad05-kube-api-access-mmwpw\") pod \"payload-processing-7bb9489676-chklw\" (UID: \"20bf40f5-8b81-478e-9e5b-c0fa554aad05\") " pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.233954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.233843 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8c4e07-e240-416d-a7ca-ddf5c91b9d54-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-42bpn\" (UID: \"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.233954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.233908 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pllmm\" (UniqueName: \"kubernetes.io/projected/8c8c4e07-e240-416d-a7ca-ddf5c91b9d54-kube-api-access-pllmm\") pod \"payload-pre-processing-7b98bb678c-42bpn\" (UID: \"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.233954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.233952 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20bf40f5-8b81-478e-9e5b-c0fa554aad05-plugins-config-volume\") pod \"payload-processing-7bb9489676-chklw\" (UID: \"20bf40f5-8b81-478e-9e5b-c0fa554aad05\") " pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.234518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.234500 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8c4e07-e240-416d-a7ca-ddf5c91b9d54-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-42bpn\" (UID: \"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.263670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.263639 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-chklw"] Nov 03 21:23:26.290544 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.290477 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pllmm\" (UniqueName: \"kubernetes.io/projected/8c8c4e07-e240-416d-a7ca-ddf5c91b9d54-kube-api-access-pllmm\") pod \"payload-pre-processing-7b98bb678c-42bpn\" (UID: \"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.334919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.334886 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20bf40f5-8b81-478e-9e5b-c0fa554aad05-plugins-config-volume\") pod \"payload-processing-7bb9489676-chklw\" (UID: \"20bf40f5-8b81-478e-9e5b-c0fa554aad05\") " pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.335064 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.334952 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmwpw\" (UniqueName: \"kubernetes.io/projected/20bf40f5-8b81-478e-9e5b-c0fa554aad05-kube-api-access-mmwpw\") pod \"payload-processing-7bb9489676-chklw\" (UID: \"20bf40f5-8b81-478e-9e5b-c0fa554aad05\") " pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.335631 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.335592 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20bf40f5-8b81-478e-9e5b-c0fa554aad05-plugins-config-volume\") pod \"payload-processing-7bb9489676-chklw\" (UID: \"20bf40f5-8b81-478e-9e5b-c0fa554aad05\") " pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.345821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.345801 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:26.395331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.395301 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmwpw\" (UniqueName: \"kubernetes.io/projected/20bf40f5-8b81-478e-9e5b-c0fa554aad05-kube-api-access-mmwpw\") pod \"payload-processing-7bb9489676-chklw\" (UID: \"20bf40f5-8b81-478e-9e5b-c0fa554aad05\") " pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.512039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.512006 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:26.715847 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.715805 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42bpn"] Nov 03 21:23:26.718490 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:23:26.718463 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8c4e07_e240_416d_a7ca_ddf5c91b9d54.slice/crio-a2ef009bc025717d64839b6f5b2bf565ff2263757b3e74a4135e4c603b37aea3 WatchSource:0}: Error finding container a2ef009bc025717d64839b6f5b2bf565ff2263757b3e74a4135e4c603b37aea3: Status 404 returned error can't find the container with id a2ef009bc025717d64839b6f5b2bf565ff2263757b3e74a4135e4c603b37aea3 Nov 03 21:23:26.791573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.791542 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" event={"ID":"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54","Type":"ContainerStarted","Data":"a2ef009bc025717d64839b6f5b2bf565ff2263757b3e74a4135e4c603b37aea3"} Nov 03 21:23:26.881370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:26.881339 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-chklw"] Nov 03 21:23:26.884636 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:23:26.884582 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bf40f5_8b81_478e_9e5b_c0fa554aad05.slice/crio-72688eca0b922d7b942a9e398fc065e60758ce02a2ba020f0f7b426f6d569e5c WatchSource:0}: Error finding container 72688eca0b922d7b942a9e398fc065e60758ce02a2ba020f0f7b426f6d569e5c: Status 404 returned error can't find the container with id 72688eca0b922d7b942a9e398fc065e60758ce02a2ba020f0f7b426f6d569e5c Nov 03 21:23:27.801582 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:27.801537 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-chklw" event={"ID":"20bf40f5-8b81-478e-9e5b-c0fa554aad05","Type":"ContainerStarted","Data":"72688eca0b922d7b942a9e398fc065e60758ce02a2ba020f0f7b426f6d569e5c"} Nov 03 21:23:29.813703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:29.813597 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" event={"ID":"8c8c4e07-e240-416d-a7ca-ddf5c91b9d54","Type":"ContainerStarted","Data":"600bf26ecf56a3c59e38d67d5f77319a4fb9b51b0b0e6217d75e4bebfbf44f08"} Nov 03 21:23:29.814104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:29.813707 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:29.815309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:29.815286 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-chklw" event={"ID":"20bf40f5-8b81-478e-9e5b-c0fa554aad05","Type":"ContainerStarted","Data":"df120d4bfff30b6426ff563bbef7836ec4ee5f73cfe5036d1a6143ba20233e76"} Nov 03 21:23:29.815371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:29.815344 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:29.925932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:29.925736 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" podStartSLOduration=2.247647675 podStartE2EDuration="4.925721193s" totalImagesPullingTime="2.678073518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:29.925519749 +0000 UTC m=+477.536367564" watchObservedRunningTime="2026-11-03 21:23:29.925721193 +0000 UTC m=+477.536569008" Nov 03 21:23:30.036962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:30.036803 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-chklw" podStartSLOduration=1.5335982750000001 podStartE2EDuration="4.036786812s" totalImagesPullingTime="2.503188537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:30.035603252 +0000 UTC m=+477.646451068" watchObservedRunningTime="2026-11-03 21:23:30.036786812 +0000 UTC m=+477.647634629" Nov 03 21:23:33.066593 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:23:33.066550 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:23:38.251421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:38.251373 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:38.477381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:38.477347 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:40.823827 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:40.823800 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-chklw" Nov 03 21:23:40.824257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:40.823869 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42bpn" Nov 03 21:23:48.293297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:48.293262 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:48.834958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:48.834925 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:50.114309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:50.114280 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:50.392866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:50.392775 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:57.048255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.048224 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz"] Nov 03 21:23:57.051743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.051726 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.106520 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.106490 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 21:23:57.112050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.112017 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.112192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.112071 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.112192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.112104 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/738cd3d2-8890-421b-bb39-38f62d78d853-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.112192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.112131 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.112192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.112161 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn44d\" (UniqueName: \"kubernetes.io/projected/738cd3d2-8890-421b-bb39-38f62d78d853-kube-api-access-wn44d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.112192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.112187 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.127362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.127336 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz"] Nov 03 21:23:57.155731 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.155707 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x8lxr\"" Nov 03 21:23:57.155854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.155707 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 21:23:57.155854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.155768 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 21:23:57.212653 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.212606 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.212799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.212689 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.212799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.212718 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/738cd3d2-8890-421b-bb39-38f62d78d853-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.212799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.212751 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.212799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.212796 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn44d\" (UniqueName: \"kubernetes.io/projected/738cd3d2-8890-421b-bb39-38f62d78d853-kube-api-access-wn44d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.213013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.212828 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.213013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.212987 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.213116 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.213074 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.213188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.213157 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.215046 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.215025 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/738cd3d2-8890-421b-bb39-38f62d78d853-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.215306 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.215289 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/738cd3d2-8890-421b-bb39-38f62d78d853-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.260665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.260633 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:57.269115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.269087 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn44d\" (UniqueName: \"kubernetes.io/projected/738cd3d2-8890-421b-bb39-38f62d78d853-kube-api-access-wn44d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz\" (UID: \"738cd3d2-8890-421b-bb39-38f62d78d853\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.363804 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.363772 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:23:57.733681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.733594 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz"] Nov 03 21:23:57.737496 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:23:57.737469 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738cd3d2_8890_421b_bb39_38f62d78d853.slice/crio-6f03c64dc9dacde22a16e8099761fed91195a29e7f80d5821c5e3db5f988e11c WatchSource:0}: Error finding container 6f03c64dc9dacde22a16e8099761fed91195a29e7f80d5821c5e3db5f988e11c: Status 404 returned error can't find the container with id 6f03c64dc9dacde22a16e8099761fed91195a29e7f80d5821c5e3db5f988e11c Nov 03 21:23:57.738755 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.738735 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:23:57.941383 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:57.941352 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" event={"ID":"738cd3d2-8890-421b-bb39-38f62d78d853","Type":"ContainerStarted","Data":"6f03c64dc9dacde22a16e8099761fed91195a29e7f80d5821c5e3db5f988e11c"} Nov 03 21:23:59.953777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:23:59.953738 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" event={"ID":"738cd3d2-8890-421b-bb39-38f62d78d853","Type":"ContainerStarted","Data":"d650f0682f4537021bd75dd38d7554e348335a5f1bd4addcc5f7e6f570c13da2"} Nov 03 21:24:00.068292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:00.068172 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" podStartSLOduration=0.989186371 podStartE2EDuration="3.068153392s" totalImagesPullingTime="2.078967021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:00.066831567 +0000 UTC m=+507.677679382" watchObservedRunningTime="2026-11-03 21:24:00.068153392 +0000 UTC m=+507.679001202" Nov 03 21:24:00.364914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:00.364882 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:24:03.067220 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:24:03.067178 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:24:03.755169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:03.755133 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:06.504279 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.504251 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95"] Nov 03 21:24:06.508563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.508545 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.567927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.567892 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95"] Nov 03 21:24:06.594848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.594814 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.594848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.594845 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.595060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.594866 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462249ac-4a5a-486f-a417-553aa37c719b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.595104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.595076 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbfdh\" (UniqueName: \"kubernetes.io/projected/462249ac-4a5a-486f-a417-553aa37c719b-kube-api-access-bbfdh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.595293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.595263 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.595386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.595371 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.615249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.615229 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 21:24:06.696061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696025 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696058 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696078 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462249ac-4a5a-486f-a417-553aa37c719b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696102 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbfdh\" (UniqueName: \"kubernetes.io/projected/462249ac-4a5a-486f-a417-553aa37c719b-kube-api-access-bbfdh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696136 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696186 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696515 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696551 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.696674 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.696585 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.698469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.698446 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462249ac-4a5a-486f-a417-553aa37c719b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.698857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.698836 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462249ac-4a5a-486f-a417-553aa37c719b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.756848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.756778 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbfdh\" (UniqueName: \"kubernetes.io/projected/462249ac-4a5a-486f-a417-553aa37c719b-kube-api-access-bbfdh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7m95\" (UID: \"462249ac-4a5a-486f-a417-553aa37c719b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.822083 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.822041 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:06.900329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:06.900296 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:07.191021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:07.190997 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95"] Nov 03 21:24:07.364151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:07.364116 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:24:07.376751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:07.376727 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:24:07.999258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:07.999215 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" event={"ID":"462249ac-4a5a-486f-a417-553aa37c719b","Type":"ContainerStarted","Data":"c42fa9147061417e65b96b8fcf929cb29ba4cbaf11e14a7250438b01d09d1381"} Nov 03 21:24:07.999749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:07.999279 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" event={"ID":"462249ac-4a5a-486f-a417-553aa37c719b","Type":"ContainerStarted","Data":"75be47cdd0a597d9716d0c2d41adeba22644c6258ad01db6f51921b11d1bd733"} Nov 03 21:24:08.010435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:08.010414 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz" Nov 03 21:24:08.112893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:08.112840 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" podStartSLOduration=1.513054711 podStartE2EDuration="2.112827328s" totalImagesPullingTime="599.772617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:08.110811829 +0000 UTC m=+515.721659645" watchObservedRunningTime="2026-11-03 21:24:08.112827328 +0000 UTC m=+515.723675172" Nov 03 21:24:08.822570 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:08.822533 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:12.312472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:12.312439 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:15.400171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:15.400114 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:16.823068 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:16.823035 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:16.835329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:16.835305 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:17.054161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:17.054136 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7m95" Nov 03 21:24:18.580293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.580257 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hxsvp"] Nov 03 21:24:18.583482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.583463 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.695268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.695231 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 21:24:18.712156 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.712122 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.712370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.712347 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.712442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.712415 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-home\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.712508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.712443 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.712508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.712474 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf84aaa-c653-42c2-9c51-eeca59ecee33-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.712637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.712535 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnprj\" (UniqueName: \"kubernetes.io/projected/fdf84aaa-c653-42c2-9c51-eeca59ecee33-kube-api-access-rnprj\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.713031 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.713008 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hxsvp"] Nov 03 21:24:18.770431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.770400 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:18.813362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.813331 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf84aaa-c653-42c2-9c51-eeca59ecee33-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.813534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.813381 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnprj\" (UniqueName: \"kubernetes.io/projected/fdf84aaa-c653-42c2-9c51-eeca59ecee33-kube-api-access-rnprj\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.813534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.813446 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.813534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.813467 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.813534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.813501 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-home\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.813534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.813527 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.813994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.813972 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.814066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.814031 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.814066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.814056 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-home\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.815711 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.815692 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf84aaa-c653-42c2-9c51-eeca59ecee33-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.816350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.816336 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf84aaa-c653-42c2-9c51-eeca59ecee33-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.872780 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.872747 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnprj\" (UniqueName: \"kubernetes.io/projected/fdf84aaa-c653-42c2-9c51-eeca59ecee33-kube-api-access-rnprj\") pod \"echo-simulator-kserve-64b6f8f96c-hxsvp\" (UID: \"fdf84aaa-c653-42c2-9c51-eeca59ecee33\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:18.896500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:18.896470 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:19.268863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:19.268789 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hxsvp"] Nov 03 21:24:19.272904 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:24:19.272877 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf84aaa_c653_42c2_9c51_eeca59ecee33.slice/crio-b6cbc5eabdaf1732b78c17f6f6e84ed2cc7145b520499b40a5367f95632b2da6 WatchSource:0}: Error finding container b6cbc5eabdaf1732b78c17f6f6e84ed2cc7145b520499b40a5367f95632b2da6: Status 404 returned error can't find the container with id b6cbc5eabdaf1732b78c17f6f6e84ed2cc7145b520499b40a5367f95632b2da6 Nov 03 21:24:20.059288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:20.059253 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" event={"ID":"fdf84aaa-c653-42c2-9c51-eeca59ecee33","Type":"ContainerStarted","Data":"b6cbc5eabdaf1732b78c17f6f6e84ed2cc7145b520499b40a5367f95632b2da6"} Nov 03 21:24:22.070528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:22.070444 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" event={"ID":"fdf84aaa-c653-42c2-9c51-eeca59ecee33","Type":"ContainerStarted","Data":"f50648838c89f1a81154be61f403d76d9b09e5da0c8f0809a17fff4cf985c615"} Nov 03 21:24:22.138975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:22.138936 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:22.185886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:22.185831 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" podStartSLOduration=1.7939220009999999 podStartE2EDuration="4.185817796s" totalImagesPullingTime="2.391895795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:22.183556419 +0000 UTC m=+529.794404234" watchObservedRunningTime="2026-11-03 21:24:22.185817796 +0000 UTC m=+529.796665656" Nov 03 21:24:22.897660 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:22.897597 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:25.518099 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.518066 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l"] Nov 03 21:24:25.521526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.521509 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.575695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.575658 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.575856 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.575797 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0fd94-4550-4065-b573-db223a11f0ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.575856 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.575847 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.575950 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.575892 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnknj\" (UniqueName: \"kubernetes.io/projected/d2e0fd94-4550-4065-b573-db223a11f0ac-kube-api-access-vnknj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.576033 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.576013 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.576098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.576084 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.622733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.622693 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 21:24:25.624596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.624565 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l"] Nov 03 21:24:25.677019 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.676974 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677053 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677112 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677143 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0fd94-4550-4065-b573-db223a11f0ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677172 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677194 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnknj\" (UniqueName: \"kubernetes.io/projected/d2e0fd94-4550-4065-b573-db223a11f0ac-kube-api-access-vnknj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677452 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677491 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.677588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.677550 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.679513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.679494 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e0fd94-4550-4065-b573-db223a11f0ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.679886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.679863 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e0fd94-4550-4065-b573-db223a11f0ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.733529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.733498 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnknj\" (UniqueName: \"kubernetes.io/projected/d2e0fd94-4550-4065-b573-db223a11f0ac-kube-api-access-vnknj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l\" (UID: \"d2e0fd94-4550-4065-b573-db223a11f0ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:25.802760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.802639 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:25.832574 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:25.832533 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:26.206658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:26.206634 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l"] Nov 03 21:24:26.208992 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:24:26.208957 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2e0fd94_4550_4065_b573_db223a11f0ac.slice/crio-a32d4492b965ab91b61bff0b27af77398677da349950bd15d8613facfb3db928 WatchSource:0}: Error finding container a32d4492b965ab91b61bff0b27af77398677da349950bd15d8613facfb3db928: Status 404 returned error can't find the container with id a32d4492b965ab91b61bff0b27af77398677da349950bd15d8613facfb3db928 Nov 03 21:24:27.096451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:27.096396 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" event={"ID":"d2e0fd94-4550-4065-b573-db223a11f0ac","Type":"ContainerStarted","Data":"12ea0d730f3799d1c87b9fb6e36e7dea9977d9388e0d3ead962882bf3dd8166e"} Nov 03 21:24:27.096451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:27.096456 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" event={"ID":"d2e0fd94-4550-4065-b573-db223a11f0ac","Type":"ContainerStarted","Data":"a32d4492b965ab91b61bff0b27af77398677da349950bd15d8613facfb3db928"} Nov 03 21:24:27.210162 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:27.210105 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podStartSLOduration=1.593051907 podStartE2EDuration="2.210088846s" totalImagesPullingTime="617.036939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:27.208533923 +0000 UTC m=+534.819381740" watchObservedRunningTime="2026-11-03 21:24:27.210088846 +0000 UTC m=+534.820936671" Nov 03 21:24:27.833655 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:27.833601 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:28.374870 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.374835 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj"] Nov 03 21:24:28.378454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.378437 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.430145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.430111 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 21:24:28.479703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.479669 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mc2lq\"" Nov 03 21:24:28.482236 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.482203 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj"] Nov 03 21:24:28.499992 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.499965 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.499999 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.500092 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.500115 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.500148 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.500178 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.500349 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.500458 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.500541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.500518 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6pw\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-kube-api-access-9m6pw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.601910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.601869 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.601910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.601909 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.601939 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.601961 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.601984 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602003 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602034 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602102 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602124 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6pw\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-kube-api-access-9m6pw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602558 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602417 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602558 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602435 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602558 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602542 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602806 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.602962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.602901 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.604376 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.604356 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.604487 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.604473 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.658981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.658915 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.658981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.658960 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6pw\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-kube-api-access-9m6pw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c8qjj\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.690964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.690929 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:28.897057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.897026 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:28.909525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:28.909455 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:29.117724 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:29.117696 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hxsvp" Nov 03 21:24:29.179661 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:29.179635 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj"] Nov 03 21:24:29.182259 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:24:29.182232 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b7fe9e_df1c_4fce_bd82_0a8acaf735b1.slice/crio-29ff9169c186c58817d9d107b410a4e372064c1de6840a238964046ff5e1f566 WatchSource:0}: Error finding container 29ff9169c186c58817d9d107b410a4e372064c1de6840a238964046ff5e1f566: Status 404 returned error can't find the container with id 29ff9169c186c58817d9d107b410a4e372064c1de6840a238964046ff5e1f566 Nov 03 21:24:30.113430 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:30.113391 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" event={"ID":"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1","Type":"ContainerStarted","Data":"29ff9169c186c58817d9d107b410a4e372064c1de6840a238964046ff5e1f566"} Nov 03 21:24:32.972968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:32.972935 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:24:32.973260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:32.973005 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:24:32.973260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:32.973034 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:24:33.068251 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:24:33.068210 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:24:33.132749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:33.132711 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" event={"ID":"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1","Type":"ContainerStarted","Data":"227b71d1ea52a8ffc3ccd8b3a4e421f11c9623976724d26f41ee31be58f8665a"} Nov 03 21:24:33.256928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:33.256812 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podStartSLOduration=1.469210332 podStartE2EDuration="5.25679348s" totalImagesPullingTime="3.787583148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:33.255101801 +0000 UTC m=+540.865949616" watchObservedRunningTime="2026-11-03 21:24:33.25679348 +0000 UTC m=+540.867641298" Nov 03 21:24:33.691665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:33.691598 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:34.693181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:34.693142 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:24:34.693666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:34.693208 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:24:35.691905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:35.691868 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:24:35.692097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:35.691923 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:24:35.843635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:35.843552 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:35.857341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:35.857311 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:36.170891 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:36.170860 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:24:36.691535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:36.691500 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:24:36.691743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:36.691558 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:24:37.692209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:37.692170 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:24:37.692600 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:37.692226 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:24:38.569001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:38.568966 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:38.691728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:38.691684 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:24:38.691926 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:38.691762 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:24:39.691961 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:39.691915 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:24:39.692374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:39.691988 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:24:39.752795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:39.752760 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:39.980600 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:39.980506 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:41.731886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:41.731777 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:24:41.731886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:41.731850 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:24:42.732853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:42.732813 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c8qjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:24:42.733268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:42.732890 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:24:42.753803 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:42.753774 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:42.754208 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:42.754183 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:42.754978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:42.754961 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:24:43.507320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.507292 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql"] Nov 03 21:24:43.510664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.510647 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.600103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.600068 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql"] Nov 03 21:24:43.649419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649383 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649424 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649439 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a40b3c2-8230-4946-a555-132a0de19452-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649457 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649504 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a40b3c2-8230-4946-a555-132a0de19452-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649657 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649690 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a40b3c2-8230-4946-a555-132a0de19452-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649730 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.649850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.649797 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsdqf\" (UniqueName: \"kubernetes.io/projected/5a40b3c2-8230-4946-a555-132a0de19452-kube-api-access-bsdqf\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.750852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.750821 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.750852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.750854 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsdqf\" (UniqueName: \"kubernetes.io/projected/5a40b3c2-8230-4946-a555-132a0de19452-kube-api-access-bsdqf\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.750909 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.750933 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.750949 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a40b3c2-8230-4946-a555-132a0de19452-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.750967 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.750981 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a40b3c2-8230-4946-a555-132a0de19452-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.751011 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.751049 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a40b3c2-8230-4946-a555-132a0de19452-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.751312 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.751337 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.751480 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.751562 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.751835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.751788 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a40b3c2-8230-4946-a555-132a0de19452-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.753397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.753376 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a40b3c2-8230-4946-a555-132a0de19452-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.753477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.753402 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a40b3c2-8230-4946-a555-132a0de19452-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.807274 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.807207 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsdqf\" (UniqueName: \"kubernetes.io/projected/5a40b3c2-8230-4946-a555-132a0de19452-kube-api-access-bsdqf\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.807274 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.807221 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a40b3c2-8230-4946-a555-132a0de19452-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b4xql\" (UID: \"5a40b3c2-8230-4946-a555-132a0de19452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:43.823108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:43.823086 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:44.246192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:44.246171 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql"] Nov 03 21:24:44.248488 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:24:44.248461 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a40b3c2_8230_4946_a555_132a0de19452.slice/crio-e815ffefd42eb52bc210b0666269f86232769a7596d0597408f643f8b2933e69 WatchSource:0}: Error finding container e815ffefd42eb52bc210b0666269f86232769a7596d0597408f643f8b2933e69: Status 404 returned error can't find the container with id e815ffefd42eb52bc210b0666269f86232769a7596d0597408f643f8b2933e69 Nov 03 21:24:44.251447 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:44.251417 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:24:44.251541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:44.251477 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:24:44.251541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:44.251502 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:24:45.212737 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:45.212691 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" event={"ID":"5a40b3c2-8230-4946-a555-132a0de19452","Type":"ContainerStarted","Data":"3b85473d6917a416011a93b123b46846ea95f9e7424240c8ec40990ffdbe7a06"} Nov 03 21:24:45.213163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:45.212757 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" event={"ID":"5a40b3c2-8230-4946-a555-132a0de19452","Type":"ContainerStarted","Data":"e815ffefd42eb52bc210b0666269f86232769a7596d0597408f643f8b2933e69"} Nov 03 21:24:45.842638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:45.835324 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:45.842965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:45.842928 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:24:45.843163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:45.843137 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:24:46.840637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:46.836716 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:24:46.840637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:46.836781 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:24:47.842638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:47.836045 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:24:47.842638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:47.836112 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:24:48.835639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:48.826038 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:24:48.835639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:48.826106 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:24:49.838689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:49.826954 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:24:49.838689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:49.827014 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:24:50.824140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:50.824110 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:24:50.824310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:50.824159 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:24:52.147403 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:52.147371 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:52.147890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:52.147767 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:52.148431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:52.148414 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" Nov 03 21:24:52.311434 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:52.311387 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podStartSLOduration=9.311375825 podStartE2EDuration="9.311375825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:45.436023283 +0000 UTC m=+553.046871113" watchObservedRunningTime="2026-11-03 21:24:52.311375825 +0000 UTC m=+559.922223635" Nov 03 21:24:53.147362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.147328 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-585df4d986-v4wk5"] Nov 03 21:24:53.150541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.150521 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.205835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.205813 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-585df4d986-v4wk5"] Nov 03 21:24:53.241823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.241760 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdw9\" (UniqueName: \"kubernetes.io/projected/02d38050-82f2-4ec5-a746-2a442440d913-kube-api-access-hsdw9\") pod \"authorino-585df4d986-v4wk5\" (UID: \"02d38050-82f2-4ec5-a746-2a442440d913\") " pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.241923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.241863 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02d38050-82f2-4ec5-a746-2a442440d913-tls-cert\") pod \"authorino-585df4d986-v4wk5\" (UID: \"02d38050-82f2-4ec5-a746-2a442440d913\") " pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.342573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.342548 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdw9\" (UniqueName: \"kubernetes.io/projected/02d38050-82f2-4ec5-a746-2a442440d913-kube-api-access-hsdw9\") pod \"authorino-585df4d986-v4wk5\" (UID: \"02d38050-82f2-4ec5-a746-2a442440d913\") " pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.342718 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.342636 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02d38050-82f2-4ec5-a746-2a442440d913-tls-cert\") pod \"authorino-585df4d986-v4wk5\" (UID: \"02d38050-82f2-4ec5-a746-2a442440d913\") " pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.345152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.345134 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02d38050-82f2-4ec5-a746-2a442440d913-tls-cert\") pod \"authorino-585df4d986-v4wk5\" (UID: \"02d38050-82f2-4ec5-a746-2a442440d913\") " pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.398409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.398390 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdw9\" (UniqueName: \"kubernetes.io/projected/02d38050-82f2-4ec5-a746-2a442440d913-kube-api-access-hsdw9\") pod \"authorino-585df4d986-v4wk5\" (UID: \"02d38050-82f2-4ec5-a746-2a442440d913\") " pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.462118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.462096 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-585df4d986-v4wk5" Nov 03 21:24:53.836104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.836037 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:53.836838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:53.836816 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-585df4d986-v4wk5"] Nov 03 21:24:53.839185 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:24:53.839163 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d38050_82f2_4ec5_a746_2a442440d913.slice/crio-d303fd73845db80ff6675abc28fa743493c1bb863af736111b205e398b2f57b8 WatchSource:0}: Error finding container d303fd73845db80ff6675abc28fa743493c1bb863af736111b205e398b2f57b8: Status 404 returned error can't find the container with id d303fd73845db80ff6675abc28fa743493c1bb863af736111b205e398b2f57b8 Nov 03 21:24:54.073229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:54.073200 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:54.277239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:54.277212 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-585df4d986-v4wk5" event={"ID":"02d38050-82f2-4ec5-a746-2a442440d913","Type":"ContainerStarted","Data":"d303fd73845db80ff6675abc28fa743493c1bb863af736111b205e398b2f57b8"} Nov 03 21:24:55.284455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.284423 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-585df4d986-v4wk5" event={"ID":"02d38050-82f2-4ec5-a746-2a442440d913","Type":"ContainerStarted","Data":"6b4d1566b04096f82e0d5f8a241be0bd5083a2a23b723ed2a58949fde9797175"} Nov 03 21:24:55.397948 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.397901 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-585df4d986-v4wk5" podStartSLOduration=1.04786157 podStartE2EDuration="2.39788745s" totalImagesPullingTime="1.35002588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:55.395733179 +0000 UTC m=+563.006580994" watchObservedRunningTime="2026-11-03 21:24:55.39788745 +0000 UTC m=+563.008735264" Nov 03 21:24:55.412409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.412383 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:55.424190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.424163 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5867475f65-zr2tb"] Nov 03 21:24:55.424399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.424370 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5867475f65-zr2tb" podUID="1a4b582e-b35d-47c2-b017-097ede55a4ff" containerName="authorino" containerID="cri-o://be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3" gracePeriod=30 Nov 03 21:24:55.652428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.652386 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:55.665378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.665356 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:24:55.765502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.765427 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a4b582e-b35d-47c2-b017-097ede55a4ff-tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4b582e-b35d-47c2-b017-097ede55a4ff-tls-cert\") pod \"1a4b582e-b35d-47c2-b017-097ede55a4ff\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " Nov 03 21:24:55.765675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.765562 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4b582e-b35d-47c2-b017-097ede55a4ff-kube-api-access-zfnnc\" (UniqueName: \"kubernetes.io/projected/1a4b582e-b35d-47c2-b017-097ede55a4ff-kube-api-access-zfnnc\") pod \"1a4b582e-b35d-47c2-b017-097ede55a4ff\" (UID: \"1a4b582e-b35d-47c2-b017-097ede55a4ff\") " Nov 03 21:24:55.767673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.767640 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4b582e-b35d-47c2-b017-097ede55a4ff-kube-api-access-zfnnc" pod "1a4b582e-b35d-47c2-b017-097ede55a4ff" (UID: "1a4b582e-b35d-47c2-b017-097ede55a4ff"). InnerVolumeSpecName "kube-api-access-zfnnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:55.779073 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.779044 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4b582e-b35d-47c2-b017-097ede55a4ff-tls-cert" pod "1a4b582e-b35d-47c2-b017-097ede55a4ff" (UID: "1a4b582e-b35d-47c2-b017-097ede55a4ff"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:55.866746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.866710 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfnnc\" (UniqueName: \"kubernetes.io/projected/1a4b582e-b35d-47c2-b017-097ede55a4ff-kube-api-access-zfnnc\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:55.866746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:55.866733 5931 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4b582e-b35d-47c2-b017-097ede55a4ff-tls-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:56.291038 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.291005 5931 generic.go:356] "Generic (PLEG): container finished" podID="1a4b582e-b35d-47c2-b017-097ede55a4ff" containerID="be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3" exitCode=0 Nov 03 21:24:56.291445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.291057 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5867475f65-zr2tb" Nov 03 21:24:56.291445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.291095 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5867475f65-zr2tb" event={"ID":"1a4b582e-b35d-47c2-b017-097ede55a4ff","Type":"ContainerDied","Data":"be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3"} Nov 03 21:24:56.291445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.291144 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5867475f65-zr2tb" event={"ID":"1a4b582e-b35d-47c2-b017-097ede55a4ff","Type":"ContainerDied","Data":"6f478c684d0991fc6b4a8b6112049e5701d55686d583c9be4b407f41cc88fc4a"} Nov 03 21:24:56.291445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.291168 5931 scope.go:122] "RemoveContainer" containerID="be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3" Nov 03 21:24:56.301962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.301944 5931 scope.go:122] "RemoveContainer" containerID="be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3" Nov 03 21:24:56.302199 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:24:56.302184 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3\": container with ID starting with be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3 not found: ID does not exist" containerID="be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3" Nov 03 21:24:56.302256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.302207 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3"} err="failed to get container status \"be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3\": rpc error: code = NotFound desc = could not find container \"be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3\": container with ID starting with be967c3232968f30ce61d2f82eb56433bba1e9a2ba8e07e1768c6b0a3cbd24b3 not found: ID does not exist" Nov 03 21:24:56.455898 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.455824 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5867475f65-zr2tb"] Nov 03 21:24:56.506368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:56.506341 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5867475f65-zr2tb"] Nov 03 21:24:58.504766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:24:58.504730 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4b582e-b35d-47c2-b017-097ede55a4ff" path="/var/lib/kubelet/pods/1a4b582e-b35d-47c2-b017-097ede55a4ff/volumes" Nov 03 21:25:03.077446 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:03.077408 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:25:24.722625 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:24.722592 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:25.001556 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.001486 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:25.283421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.283348 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:25.556314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.556247 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:25.673704 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.673675 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7"] Nov 03 21:25:25.674104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.674091 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1a4b582e-b35d-47c2-b017-097ede55a4ff" containerName="authorino" Nov 03 21:25:25.674146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.674108 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4b582e-b35d-47c2-b017-097ede55a4ff" containerName="authorino" Nov 03 21:25:25.674209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.674200 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4b582e-b35d-47c2-b017-097ede55a4ff" containerName="authorino" Nov 03 21:25:25.677632 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.677587 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.729419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.729380 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-889495a7-openshift-default-dockercfg-62jlh\"" Nov 03 21:25:25.780035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.780004 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7"] Nov 03 21:25:25.830339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830268 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d74c17-7790-4833-8cfa-06fcbec02964-istiod-ca-cert\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830299 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggkf\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-kube-api-access-9ggkf\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830318 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-certs\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830486 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-envoy\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830566 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d74c17-7790-4833-8cfa-06fcbec02964-istio-podinfo\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830603 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-istio-token\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830680 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-data\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830739 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830710 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-credential-socket\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.830835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.830819 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-socket\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.833425 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.833396 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:25.932052 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932024 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d74c17-7790-4833-8cfa-06fcbec02964-istio-podinfo\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932064 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-istio-token\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932101 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-data\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932136 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-credential-socket\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932167 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-socket\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932193 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d74c17-7790-4833-8cfa-06fcbec02964-istiod-ca-cert\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932222 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggkf\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-kube-api-access-9ggkf\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932462 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932250 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-certs\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.932462 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.932327 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-envoy\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.933363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.933318 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-credential-socket\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.933363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.933348 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-data\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.933556 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.933535 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-certs\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.933962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.933939 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d74c17-7790-4833-8cfa-06fcbec02964-istiod-ca-cert\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.934222 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.934200 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-socket\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.935116 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.935090 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-envoy\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.935288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.935269 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d74c17-7790-4833-8cfa-06fcbec02964-istio-podinfo\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.988478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.988457 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-istio-token\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.988478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.988475 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggkf\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-kube-api-access-9ggkf\") pod \"e2e-ait-889495a7-openshift-default-d4df596ff-h64h7\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:25.991430 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:25.991412 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:26.061494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.061464 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:26.301810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.301775 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:26.416603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.416569 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7"] Nov 03 21:25:26.423333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.423289 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:26.423432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.423362 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:26.423432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.423393 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:26.447511 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.447488 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" event={"ID":"37d74c17-7790-4833-8cfa-06fcbec02964","Type":"ContainerStarted","Data":"7462ad65e164b06cda74c3369056271a879ceabfdd43500bc712d40a65f19c42"} Nov 03 21:25:26.522168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.522135 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:26.758864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:26.758835 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:27.454110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:27.454077 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" event={"ID":"37d74c17-7790-4833-8cfa-06fcbec02964","Type":"ContainerStarted","Data":"d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349"} Nov 03 21:25:27.619745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:27.619692 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" podStartSLOduration=2.619678356 podStartE2EDuration="2.619678356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:27.61803945 +0000 UTC m=+595.228887266" watchObservedRunningTime="2026-11-03 21:25:27.619678356 +0000 UTC m=+595.230526170" Nov 03 21:25:27.991941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:27.991904 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:27.996964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:27.996938 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:28.176054 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:28.176023 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:28.459592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:28.459556 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:28.460412 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:28.460397 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:28.498884 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:28.498854 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:29.156742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:29.156716 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:29.501220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:29.501146 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:29.959976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:29.959944 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:30.217060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:30.216994 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:31.792217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:31.792189 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:32.031485 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.031456 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:32.272860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.272824 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:32.517911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.517883 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:32.532411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.532358 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj"] Nov 03 21:25:32.536709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.536693 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.588368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.588342 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:25:32.591263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.591228 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:25:32.609105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.609087 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-daf21c29-openshift-default-dockercfg-kgckm\"" Nov 03 21:25:32.642802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.642781 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj"] Nov 03 21:25:32.710308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710284 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-token\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710318 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istiod-ca-cert\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710487 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710403 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-certs\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710487 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710450 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-socket\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710507 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-credential-socket\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710586 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqn2\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-kube-api-access-2xqn2\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710709 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-data\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710800 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710750 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-podinfo\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.710800 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.710788 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-envoy\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.772796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.772774 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:32.812165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812113 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqn2\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-kube-api-access-2xqn2\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812169 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-data\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812191 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-podinfo\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812208 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-envoy\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812241 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-token\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812258 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istiod-ca-cert\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812442 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-certs\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812493 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-socket\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812517 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-credential-socket\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.812949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.812602 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-data\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.813056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.813025 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-socket\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.813056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.813046 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-certs\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.813189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.813171 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-credential-socket\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.813384 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.813365 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istiod-ca-cert\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.814575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.814560 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-envoy\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.815144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.815127 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-podinfo\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.871651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.870359 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqn2\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-kube-api-access-2xqn2\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.884701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.884672 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-token\") pod \"e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:32.887732 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.887707 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9"] Nov 03 21:25:32.893037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.893015 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:32.913540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.913510 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec2544-7d64-4ecd-b486-6ddd16c01914-plugins-config-volume\") pod \"payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:32.913812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.913664 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpb4\" (UniqueName: \"kubernetes.io/projected/1bec2544-7d64-4ecd-b486-6ddd16c01914-kube-api-access-tkpb4\") pod \"payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:32.949401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.949155 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-35257bc0\"" Nov 03 21:25:32.949401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.949219 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-35257bc0-dockercfg-mbjgv\"" Nov 03 21:25:32.951460 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:32.951437 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9"] Nov 03 21:25:33.011808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.011780 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:33.014186 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.014167 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec2544-7d64-4ecd-b486-6ddd16c01914-plugins-config-volume\") pod \"payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:33.014290 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.014278 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpb4\" (UniqueName: \"kubernetes.io/projected/1bec2544-7d64-4ecd-b486-6ddd16c01914-kube-api-access-tkpb4\") pod \"payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:33.014828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.014812 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec2544-7d64-4ecd-b486-6ddd16c01914-plugins-config-volume\") pod \"payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:33.067192 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:33.067129 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:25:33.071249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.071227 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpb4\" (UniqueName: \"kubernetes.io/projected/1bec2544-7d64-4ecd-b486-6ddd16c01914-kube-api-access-tkpb4\") pod \"payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:33.149127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.149107 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:33.211859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.211042 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:33.575316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.575143 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj"] Nov 03 21:25:33.577636 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:25:33.577585 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9aa6f1_9f36_4ea5_950d_8e79e6e8a973.slice/crio-9db35fe09df2b3be50073ee7b0f91ede381f3b8ac9d4d7926d1d913763d78ea5 WatchSource:0}: Error finding container 9db35fe09df2b3be50073ee7b0f91ede381f3b8ac9d4d7926d1d913763d78ea5: Status 404 returned error can't find the container with id 9db35fe09df2b3be50073ee7b0f91ede381f3b8ac9d4d7926d1d913763d78ea5 Nov 03 21:25:33.579644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.579589 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:33.579734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.579678 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:33.579734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.579708 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:33.626405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:33.626385 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9"] Nov 03 21:25:33.627331 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:25:33.627305 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bec2544_7d64_4ecd_b486_6ddd16c01914.slice/crio-f12228014913127f6888722ce10f88eac2a3e11622415ac71f00d0295b2fdcb8 WatchSource:0}: Error finding container f12228014913127f6888722ce10f88eac2a3e11622415ac71f00d0295b2fdcb8: Status 404 returned error can't find the container with id f12228014913127f6888722ce10f88eac2a3e11622415ac71f00d0295b2fdcb8 Nov 03 21:25:34.132235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.132202 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:34.373561 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.373533 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:34.489554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.489483 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" event={"ID":"1bec2544-7d64-4ecd-b486-6ddd16c01914","Type":"ContainerStarted","Data":"75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951"} Nov 03 21:25:34.489704 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.489573 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" event={"ID":"1bec2544-7d64-4ecd-b486-6ddd16c01914","Type":"ContainerStarted","Data":"f12228014913127f6888722ce10f88eac2a3e11622415ac71f00d0295b2fdcb8"} Nov 03 21:25:34.489750 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.489707 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:34.491067 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.491043 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" event={"ID":"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973","Type":"ContainerStarted","Data":"8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b"} Nov 03 21:25:34.491177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.491082 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" event={"ID":"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973","Type":"ContainerStarted","Data":"9db35fe09df2b3be50073ee7b0f91ede381f3b8ac9d4d7926d1d913763d78ea5"} Nov 03 21:25:34.605078 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.605039 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" podStartSLOduration=2.605027348 podStartE2EDuration="2.605027348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:34.602698571 +0000 UTC m=+602.213546387" watchObservedRunningTime="2026-11-03 21:25:34.605027348 +0000 UTC m=+602.215875163" Nov 03 21:25:34.722330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:34.722275 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podStartSLOduration=2.722260855 podStartE2EDuration="2.722260855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:34.719775867 +0000 UTC m=+602.330623682" watchObservedRunningTime="2026-11-03 21:25:34.722260855 +0000 UTC m=+602.333108669" Nov 03 21:25:35.149530 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.149502 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:35.154366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.154344 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:35.340185 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.340161 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659"] Nov 03 21:25:35.343736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.343721 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.395975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.395938 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:25:35.396106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.396002 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:25:35.396106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.396019 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:25:35.396106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.396020 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-889495a7-serving-cert\"" Nov 03 21:25:35.398548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.398536 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nbmd6\"" Nov 03 21:25:35.401581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.401527 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659"] Nov 03 21:25:35.432057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.432035 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-metrics-certs\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.432157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.432145 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.432194 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.432171 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-openshift-service-ca\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.432265 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.432248 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4b4\" (UniqueName: \"kubernetes.io/projected/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-kube-api-access-9f4b4\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.496062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.496034 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:35.496852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.496836 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:25:35.532730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.532708 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.532852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.532742 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-openshift-service-ca\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.532852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.532795 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4b4\" (UniqueName: \"kubernetes.io/projected/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-kube-api-access-9f4b4\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.532965 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:35.532867 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-serving-cert: secret "maas-api-e2e-ait-889495a7-serving-cert" not found Nov 03 21:25:35.532965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.532879 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-metrics-certs\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.532965 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:35.532933 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls podName:755b28c1-a3d8-41c4-8241-ebd9c853dcd0 nodeName:}" failed. No retries permitted until 2026-11-03 21:25:36.032911774 +0000 UTC m=+603.643759584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls") pod "maas-api-e2e-ait-889495a7-d98959c55-lg659" (UID: "755b28c1-a3d8-41c4-8241-ebd9c853dcd0") : secret "maas-api-e2e-ait-889495a7-serving-cert" not found Nov 03 21:25:35.534401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.534376 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-openshift-service-ca\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.536295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.536272 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-metrics-certs\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.592942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.592918 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4b4\" (UniqueName: \"kubernetes.io/projected/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-kube-api-access-9f4b4\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:35.615105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.615084 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:35.855474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:35.854351 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:36.038138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:36.038108 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:36.040642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:36.040588 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls\") pod \"maas-api-e2e-ait-889495a7-d98959c55-lg659\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:36.085596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:36.085569 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:36.256332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:36.256270 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:36.628161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:36.628139 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659"] Nov 03 21:25:36.632498 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:25:36.632471 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755b28c1_a3d8_41c4_8241_ebd9c853dcd0.slice/crio-d9a314954e55646ba57dc274879530bc39365f88e45966b0e2e440f943aeec6a WatchSource:0}: Error finding container d9a314954e55646ba57dc274879530bc39365f88e45966b0e2e440f943aeec6a: Status 404 returned error can't find the container with id d9a314954e55646ba57dc274879530bc39365f88e45966b0e2e440f943aeec6a Nov 03 21:25:37.353775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.353745 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:37.508558 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.508523 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" event={"ID":"755b28c1-a3d8-41c4-8241-ebd9c853dcd0","Type":"ContainerStarted","Data":"d9a314954e55646ba57dc274879530bc39365f88e45966b0e2e440f943aeec6a"} Nov 03 21:25:37.587126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.587090 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:37.815204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.815133 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t"] Nov 03 21:25:37.818822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.818803 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:37.855456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.855427 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-plugins-config-volume\") pod \"payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:37.855585 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.855512 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcfxh\" (UniqueName: \"kubernetes.io/projected/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-kube-api-access-wcfxh\") pod \"payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:37.921205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.921180 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-889495a7\"" Nov 03 21:25:37.921205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.921221 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-889495a7-dockercfg-qtbbp\"" Nov 03 21:25:37.923136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.923116 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t"] Nov 03 21:25:37.955984 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.955960 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcfxh\" (UniqueName: \"kubernetes.io/projected/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-kube-api-access-wcfxh\") pod \"payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:37.956079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.956046 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-plugins-config-volume\") pod \"payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:37.956676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:37.956659 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-plugins-config-volume\") pod \"payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:38.012924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.012900 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcfxh\" (UniqueName: \"kubernetes.io/projected/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-kube-api-access-wcfxh\") pod \"payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:38.131063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.131045 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:38.503231 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:25:38.503204 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10650a0_df9f_45f3_b49f_60b0ba5eb47c.slice/crio-5df4738a569ade65fb7a85a9e655183cb32b3312c9103459f2264ee0d895d507 WatchSource:0}: Error finding container 5df4738a569ade65fb7a85a9e655183cb32b3312c9103459f2264ee0d895d507: Status 404 returned error can't find the container with id 5df4738a569ade65fb7a85a9e655183cb32b3312c9103459f2264ee0d895d507 Nov 03 21:25:38.506191 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.506171 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t"] Nov 03 21:25:38.515550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.515519 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" event={"ID":"755b28c1-a3d8-41c4-8241-ebd9c853dcd0","Type":"ContainerStarted","Data":"8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758"} Nov 03 21:25:38.515655 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.515599 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:38.517720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.517696 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" event={"ID":"d10650a0-df9f-45f3-b49f-60b0ba5eb47c","Type":"ContainerStarted","Data":"5df4738a569ade65fb7a85a9e655183cb32b3312c9103459f2264ee0d895d507"} Nov 03 21:25:38.671547 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.671493 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" podStartSLOduration=2.765851208 podStartE2EDuration="3.671478708s" totalImagesPullingTime="905.6275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:38.670433949 +0000 UTC m=+606.281281763" watchObservedRunningTime="2026-11-03 21:25:38.671478708 +0000 UTC m=+606.282326523" Nov 03 21:25:38.908975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:38.908946 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:39.524183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:39.524149 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" event={"ID":"d10650a0-df9f-45f3-b49f-60b0ba5eb47c","Type":"ContainerStarted","Data":"38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c"} Nov 03 21:25:39.524528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:39.524279 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:39.637152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:39.637089 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" podStartSLOduration=2.637072691 podStartE2EDuration="2.637072691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:39.636342938 +0000 UTC m=+607.247190754" watchObservedRunningTime="2026-11-03 21:25:39.637072691 +0000 UTC m=+607.247920508" Nov 03 21:25:39.836301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:39.836236 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:40.083797 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.083768 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:40.393218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.393187 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:40.679973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.679905 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:40.837726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.837686 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5"] Nov 03 21:25:40.841190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.841173 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.879987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.879966 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4pm\" (UniqueName: \"kubernetes.io/projected/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-kube-api-access-5s4pm\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.880096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.880034 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.880169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.880120 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-metrics-certs\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.880169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.880160 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-openshift-service-ca\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.893238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.893213 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c2e645a4-serving-cert\"" Nov 03 21:25:40.944546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.944492 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5"] Nov 03 21:25:40.981039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.981008 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4pm\" (UniqueName: \"kubernetes.io/projected/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-kube-api-access-5s4pm\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.981135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.981095 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.981211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.981144 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-metrics-certs\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.981211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.981182 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-openshift-service-ca\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.981312 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:40.981203 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-serving-cert: secret "maas-api-e2e-ait-c2e645a4-serving-cert" not found Nov 03 21:25:40.981312 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:40.981270 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls podName:7ea3cca1-2dd6-453d-9b70-5092fc85beb5 nodeName:}" failed. No retries permitted until 2026-11-03 21:25:41.481251017 +0000 UTC m=+609.092098814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls") pod "maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" (UID: "7ea3cca1-2dd6-453d-9b70-5092fc85beb5") : secret "maas-api-e2e-ait-c2e645a4-serving-cert" not found Nov 03 21:25:40.981873 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.981845 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-openshift-service-ca\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:40.984325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:40.984305 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-metrics-certs\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:41.040146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:41.040125 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4pm\" (UniqueName: \"kubernetes.io/projected/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-kube-api-access-5s4pm\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:41.192627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:41.192580 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:41.424136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:41.424110 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:41.486248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:41.486214 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:41.488639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:41.488585 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls\") pod \"maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:41.753185 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:41.753126 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:42.128437 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:42.128416 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5"] Nov 03 21:25:42.130426 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:25:42.130400 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea3cca1_2dd6_453d_9b70_5092fc85beb5.slice/crio-85ce2208a17bc199f2bb3733b6813f1286c205e529623de13d4e412fc262cfd2 WatchSource:0}: Error finding container 85ce2208a17bc199f2bb3733b6813f1286c205e529623de13d4e412fc262cfd2: Status 404 returned error can't find the container with id 85ce2208a17bc199f2bb3733b6813f1286c205e529623de13d4e412fc262cfd2 Nov 03 21:25:42.405131 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:42.405051 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:42.538320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:42.538283 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" event={"ID":"7ea3cca1-2dd6-453d-9b70-5092fc85beb5","Type":"ContainerStarted","Data":"85ce2208a17bc199f2bb3733b6813f1286c205e529623de13d4e412fc262cfd2"} Nov 03 21:25:42.643181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:42.643148 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:43.545484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:43.545450 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" event={"ID":"7ea3cca1-2dd6-453d-9b70-5092fc85beb5","Type":"ContainerStarted","Data":"11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d"} Nov 03 21:25:43.545856 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:43.545552 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:43.660443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:43.660391 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" podStartSLOduration=2.716244504 podStartE2EDuration="3.660378964s" totalImagesPullingTime="944.13446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:43.657899445 +0000 UTC m=+611.268747259" watchObservedRunningTime="2026-11-03 21:25:43.660378964 +0000 UTC m=+611.271226814" Nov 03 21:25:44.529773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:44.529746 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:25:45.499333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:45.499309 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:25:46.157934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.157903 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv"] Nov 03 21:25:46.162331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.162307 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.217840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.217816 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv"] Nov 03 21:25:46.233224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.233202 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.233339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.233251 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-openshift-service-ca\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.233381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.233365 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-metrics-certs\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.233415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.233393 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2j6\" (UniqueName: \"kubernetes.io/projected/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-kube-api-access-7c2j6\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.263023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.263001 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-35257bc0-serving-cert\"" Nov 03 21:25:46.334399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.334374 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-metrics-certs\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.334531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.334412 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2j6\" (UniqueName: \"kubernetes.io/projected/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-kube-api-access-7c2j6\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.334531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.334458 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.334531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.334504 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-openshift-service-ca\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.334728 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:46.334586 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-serving-cert: secret "maas-api-e2e-ait-35257bc0-serving-cert" not found Nov 03 21:25:46.334728 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:25:46.334664 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls podName:1bbc622c-fcfb-41f5-a09d-999d4ade61e1 nodeName:}" failed. No retries permitted until 2026-11-03 21:25:46.834649796 +0000 UTC m=+614.445497590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls") pod "maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" (UID: "1bbc622c-fcfb-41f5-a09d-999d4ade61e1") : secret "maas-api-e2e-ait-35257bc0-serving-cert" not found Nov 03 21:25:46.335115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.335099 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-openshift-service-ca\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.336837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.336815 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-metrics-certs\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.392951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.392923 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2j6\" (UniqueName: \"kubernetes.io/projected/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-kube-api-access-7c2j6\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.448641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.448532 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:46.671858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.671827 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:46.839413 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.839351 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:46.841840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:46.841810 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls\") pod \"maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:47.077474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:47.077446 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:47.453688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:47.453600 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv"] Nov 03 21:25:47.457127 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:25:47.457097 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbc622c_fcfb_41f5_a09d_999d4ade61e1.slice/crio-2bdc3cab6f685c44d3510eb7f3cb97fd7dfed6cdf54555f6c4b3a24bf3d40fa9 WatchSource:0}: Error finding container 2bdc3cab6f685c44d3510eb7f3cb97fd7dfed6cdf54555f6c4b3a24bf3d40fa9: Status 404 returned error can't find the container with id 2bdc3cab6f685c44d3510eb7f3cb97fd7dfed6cdf54555f6c4b3a24bf3d40fa9 Nov 03 21:25:47.566911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:47.566878 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" event={"ID":"1bbc622c-fcfb-41f5-a09d-999d4ade61e1","Type":"ContainerStarted","Data":"2bdc3cab6f685c44d3510eb7f3cb97fd7dfed6cdf54555f6c4b3a24bf3d40fa9"} Nov 03 21:25:48.573908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:48.573872 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" event={"ID":"1bbc622c-fcfb-41f5-a09d-999d4ade61e1","Type":"ContainerStarted","Data":"16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3"} Nov 03 21:25:48.574269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:48.573994 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:48.706481 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:48.706383 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" podStartSLOduration=1.796048718 podStartE2EDuration="2.706369537s" totalImagesPullingTime="910.320819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:48.704583146 +0000 UTC m=+616.315430960" watchObservedRunningTime="2026-11-03 21:25:48.706369537 +0000 UTC m=+616.317217350" Nov 03 21:25:49.550912 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:49.550876 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:49.555138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:49.555118 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:25:49.776228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:49.776196 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:50.549042 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:50.549013 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:25:51.488941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:51.488910 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:51.718751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:51.718703 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:51.949189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:51.949156 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:54.585931 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:54.585901 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:25:56.147058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:56.147020 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:56.616566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:56.616534 5931 generic.go:356] "Generic (PLEG): container finished" podID="37d74c17-7790-4833-8cfa-06fcbec02964" containerID="d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349" exitCode=0 Nov 03 21:25:56.616779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:56.616636 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" event={"ID":"37d74c17-7790-4833-8cfa-06fcbec02964","Type":"ContainerDied","Data":"d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349"} Nov 03 21:25:56.617107 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:56.617083 5931 scope.go:122] "RemoveContainer" containerID="d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349" Nov 03 21:25:56.618371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:56.618347 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:56.618427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:56.618415 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:56.618465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:56.618447 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:25:57.644637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:57.643479 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" event={"ID":"37d74c17-7790-4833-8cfa-06fcbec02964","Type":"ContainerStarted","Data":"0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9"} Nov 03 21:25:57.991845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:57.991765 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:57.992969 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:57.992945 5931 patch_prober.go:29] interesting pod/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:25:57.993084 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:57.992995 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:25:58.460014 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:58.459986 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:25:58.992394 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:58.992365 5931 patch_prober.go:29] interesting pod/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:25:58.992786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:58.992410 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:25:59.992223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:59.992189 5931 patch_prober.go:29] interesting pod/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:25:59.992392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:25:59.992236 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:26:00.877833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:00.877798 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:00.992146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:00.992104 5931 patch_prober.go:29] interesting pod/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:26:00.992351 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:00.992180 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:26:01.109760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:01.109728 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:01.346635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:01.344950 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:01.992756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:01.992723 5931 patch_prober.go:29] interesting pod/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:26:01.993202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:01.993165 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:26:02.996194 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:02.996167 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:26:02.997230 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:02.997212 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:26:03.067209 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:03.067180 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:26:05.738361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:05.738325 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:10.447165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:10.447123 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:15.337783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:15.337695 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:15.745458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:15.745427 5931 generic.go:356] "Generic (PLEG): container finished" podID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerID="8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b" exitCode=0 Nov 03 21:26:15.745650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:15.745500 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" event={"ID":"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973","Type":"ContainerDied","Data":"8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b"} Nov 03 21:26:15.745893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:15.745879 5931 scope.go:122] "RemoveContainer" containerID="8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b" Nov 03 21:26:15.747097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:15.747075 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:15.747162 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:15.747141 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:15.747199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:15.747173 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:16.754400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:16.754364 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" event={"ID":"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973","Type":"ContainerStarted","Data":"43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada"} Nov 03 21:26:17.149781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:17.149744 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:26:17.151037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:17.151012 5931 patch_prober.go:29] interesting pod/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:26:17.151158 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:17.151064 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:26:18.150057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:18.150023 5931 patch_prober.go:29] interesting pod/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:26:18.150429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:18.150077 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:26:19.149576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:19.149543 5931 patch_prober.go:29] interesting pod/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:26:19.149764 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:19.149593 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:26:20.150036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:20.149999 5931 patch_prober.go:29] interesting pod/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:26:20.150521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:20.150049 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:26:20.341636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:20.341449 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:20.496667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:20.496570 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:26:21.149833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:21.149799 5931 patch_prober.go:29] interesting pod/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:26:21.150012 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:21.149859 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:26:22.152748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:22.152721 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:26:22.153707 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:22.153688 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:26:25.076209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:25.076177 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:28.148436 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:28.148404 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t"] Nov 03 21:26:28.624084 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:28.624013 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:28.865312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:28.865276 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:30.356434 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:30.356407 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:31.189108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:31.189074 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:31.420777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:31.420748 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:33.066477 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:33.066440 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:26:33.149716 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.149669 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" podUID="d10650a0-df9f-45f3-b49f-60b0ba5eb47c" containerName="payload-processing" containerID="cri-o://38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c" gracePeriod=25 Nov 03 21:26:33.398521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.398501 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:26:33.516801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.516775 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-plugins-config-volume\") pod \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " Nov 03 21:26:33.516937 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.516883 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-kube-api-access-wcfxh\" (UniqueName: \"kubernetes.io/projected/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-kube-api-access-wcfxh\") pod \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\" (UID: \"d10650a0-df9f-45f3-b49f-60b0ba5eb47c\") " Nov 03 21:26:33.517135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.517113 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-plugins-config-volume" pod "d10650a0-df9f-45f3-b49f-60b0ba5eb47c" (UID: "d10650a0-df9f-45f3-b49f-60b0ba5eb47c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:33.519354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.519327 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-kube-api-access-wcfxh" pod "d10650a0-df9f-45f3-b49f-60b0ba5eb47c" (UID: "d10650a0-df9f-45f3-b49f-60b0ba5eb47c"). InnerVolumeSpecName "kube-api-access-wcfxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:33.617503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.617480 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:33.617503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.617502 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcfxh\" (UniqueName: \"kubernetes.io/projected/d10650a0-df9f-45f3-b49f-60b0ba5eb47c-kube-api-access-wcfxh\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:33.846675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.846650 5931 generic.go:356] "Generic (PLEG): container finished" podID="d10650a0-df9f-45f3-b49f-60b0ba5eb47c" containerID="38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c" exitCode=0 Nov 03 21:26:33.846811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.846705 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" event={"ID":"d10650a0-df9f-45f3-b49f-60b0ba5eb47c","Type":"ContainerDied","Data":"38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c"} Nov 03 21:26:33.846811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.846712 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" Nov 03 21:26:33.846811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.846753 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t" event={"ID":"d10650a0-df9f-45f3-b49f-60b0ba5eb47c","Type":"ContainerDied","Data":"5df4738a569ade65fb7a85a9e655183cb32b3312c9103459f2264ee0d895d507"} Nov 03 21:26:33.846811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.846775 5931 scope.go:122] "RemoveContainer" containerID="38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c" Nov 03 21:26:33.856308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.856289 5931 scope.go:122] "RemoveContainer" containerID="38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c" Nov 03 21:26:33.856559 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:33.856542 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c\": container with ID starting with 38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c not found: ID does not exist" containerID="38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c" Nov 03 21:26:33.856653 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:33.856571 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c"} err="failed to get container status \"38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c\": rpc error: code = NotFound desc = could not find container \"38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c\": container with ID starting with 38f60ef0e8c86c02df3a4a23d957101d356989050dcc9b66942ab5023f741f8c not found: ID does not exist" Nov 03 21:26:34.009214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.009192 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t"] Nov 03 21:26:34.014552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.014531 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-889495a7-6ddd6ccb8-t555t"] Nov 03 21:26:34.504885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.504850 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10650a0-df9f-45f3-b49f-60b0ba5eb47c" path="/var/lib/kubelet/pods/d10650a0-df9f-45f3-b49f-60b0ba5eb47c/volumes" Nov 03 21:26:34.541384 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.541349 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659"] Nov 03 21:26:34.541602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.541578 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" podUID="755b28c1-a3d8-41c4-8241-ebd9c853dcd0" containerName="maas-api" containerID="cri-o://8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758" gracePeriod=30 Nov 03 21:26:34.732962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.732924 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:34.784777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.784754 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:26:34.853658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.853631 5931 generic.go:356] "Generic (PLEG): container finished" podID="755b28c1-a3d8-41c4-8241-ebd9c853dcd0" containerID="8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758" exitCode=0 Nov 03 21:26:34.853768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.853665 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" Nov 03 21:26:34.853768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.853704 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" event={"ID":"755b28c1-a3d8-41c4-8241-ebd9c853dcd0","Type":"ContainerDied","Data":"8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758"} Nov 03 21:26:34.853768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.853753 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659" event={"ID":"755b28c1-a3d8-41c4-8241-ebd9c853dcd0","Type":"ContainerDied","Data":"d9a314954e55646ba57dc274879530bc39365f88e45966b0e2e440f943aeec6a"} Nov 03 21:26:34.853900 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.853781 5931 scope.go:122] "RemoveContainer" containerID="8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758" Nov 03 21:26:34.862438 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.862424 5931 scope.go:122] "RemoveContainer" containerID="8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758" Nov 03 21:26:34.862690 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:34.862676 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758\": container with ID starting with 8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758 not found: ID does not exist" containerID="8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758" Nov 03 21:26:34.862740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.862700 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758"} err="failed to get container status \"8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758\": rpc error: code = NotFound desc = could not find container \"8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758\": container with ID starting with 8af9a5072cb36ab4947a51c5fd96651fefa289beb33045c657c2f5389c90c758 not found: ID does not exist" Nov 03 21:26:34.929278 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.929256 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-metrics-certs\") pod \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " Nov 03 21:26:34.929392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.929347 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-openshift-service-ca\") pod \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " Nov 03 21:26:34.929392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.929384 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-kube-api-access-9f4b4\" (UniqueName: \"kubernetes.io/projected/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-kube-api-access-9f4b4\") pod \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " Nov 03 21:26:34.929506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.929421 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls\") pod \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\" (UID: \"755b28c1-a3d8-41c4-8241-ebd9c853dcd0\") " Nov 03 21:26:34.929768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.929741 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-openshift-service-ca" pod "755b28c1-a3d8-41c4-8241-ebd9c853dcd0" (UID: "755b28c1-a3d8-41c4-8241-ebd9c853dcd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:34.931488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.931462 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls" pod "755b28c1-a3d8-41c4-8241-ebd9c853dcd0" (UID: "755b28c1-a3d8-41c4-8241-ebd9c853dcd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:34.931596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.931489 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-kube-api-access-9f4b4" pod "755b28c1-a3d8-41c4-8241-ebd9c853dcd0" (UID: "755b28c1-a3d8-41c4-8241-ebd9c853dcd0"). InnerVolumeSpecName "kube-api-access-9f4b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:34.931596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:34.931537 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-metrics-certs" pod "755b28c1-a3d8-41c4-8241-ebd9c853dcd0" (UID: "755b28c1-a3d8-41c4-8241-ebd9c853dcd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:35.030283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.030215 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:35.030283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.030239 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:35.030283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.030261 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:35.030283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.030271 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f4b4\" (UniqueName: \"kubernetes.io/projected/755b28c1-a3d8-41c4-8241-ebd9c853dcd0-kube-api-access-9f4b4\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:35.277980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.277944 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659"] Nov 03 21:26:35.327229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.327168 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-889495a7-d98959c55-lg659"] Nov 03 21:26:35.755699 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.755666 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5"] Nov 03 21:26:35.756082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.755871 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" podUID="7ea3cca1-2dd6-453d-9b70-5092fc85beb5" containerName="maas-api" containerID="cri-o://11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d" gracePeriod=30 Nov 03 21:26:35.950002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:35.949969 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:36.024666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.024643 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:26:36.141027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.141000 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-kube-api-access-5s4pm\" (UniqueName: \"kubernetes.io/projected/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-kube-api-access-5s4pm\") pod \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " Nov 03 21:26:36.141169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.141067 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-openshift-service-ca\") pod \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " Nov 03 21:26:36.141169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.141105 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls\") pod \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " Nov 03 21:26:36.141169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.141131 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-metrics-certs\") pod \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\" (UID: \"7ea3cca1-2dd6-453d-9b70-5092fc85beb5\") " Nov 03 21:26:36.141599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.141567 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-openshift-service-ca" pod "7ea3cca1-2dd6-453d-9b70-5092fc85beb5" (UID: "7ea3cca1-2dd6-453d-9b70-5092fc85beb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:36.143868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.143842 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-metrics-certs" pod "7ea3cca1-2dd6-453d-9b70-5092fc85beb5" (UID: "7ea3cca1-2dd6-453d-9b70-5092fc85beb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:36.143997 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.143968 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls" pod "7ea3cca1-2dd6-453d-9b70-5092fc85beb5" (UID: "7ea3cca1-2dd6-453d-9b70-5092fc85beb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:36.144114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.143933 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-kube-api-access-5s4pm" pod "7ea3cca1-2dd6-453d-9b70-5092fc85beb5" (UID: "7ea3cca1-2dd6-453d-9b70-5092fc85beb5"). InnerVolumeSpecName "kube-api-access-5s4pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:36.242753 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.242724 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s4pm\" (UniqueName: \"kubernetes.io/projected/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-kube-api-access-5s4pm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:36.242753 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.242748 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:36.242875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.242758 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:36.242875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.242766 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3cca1-2dd6-453d-9b70-5092fc85beb5-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:36.416835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.416803 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:36.504534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.504508 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755b28c1-a3d8-41c4-8241-ebd9c853dcd0" path="/var/lib/kubelet/pods/755b28c1-a3d8-41c4-8241-ebd9c853dcd0/volumes" Nov 03 21:26:36.651762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.651733 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:36.856348 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.856320 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7"] Nov 03 21:26:36.856739 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.856508 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" containerID="cri-o://0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9" gracePeriod=30 Nov 03 21:26:36.865517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.865491 5931 generic.go:356] "Generic (PLEG): container finished" podID="7ea3cca1-2dd6-453d-9b70-5092fc85beb5" containerID="11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d" exitCode=0 Nov 03 21:26:36.865636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.865557 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" event={"ID":"7ea3cca1-2dd6-453d-9b70-5092fc85beb5","Type":"ContainerDied","Data":"11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d"} Nov 03 21:26:36.865636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.865605 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" Nov 03 21:26:36.865636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.865630 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5" event={"ID":"7ea3cca1-2dd6-453d-9b70-5092fc85beb5","Type":"ContainerDied","Data":"85ce2208a17bc199f2bb3733b6813f1286c205e529623de13d4e412fc262cfd2"} Nov 03 21:26:36.865744 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.865659 5931 scope.go:122] "RemoveContainer" containerID="11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d" Nov 03 21:26:36.874500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.874480 5931 scope.go:122] "RemoveContainer" containerID="11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d" Nov 03 21:26:36.874770 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:36.874752 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d\": container with ID starting with 11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d not found: ID does not exist" containerID="11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d" Nov 03 21:26:36.874820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.874779 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d"} err="failed to get container status \"11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d\": rpc error: code = NotFound desc = could not find container \"11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d\": container with ID starting with 11edb8c389c9e07e19b85c0c2bfdde17563479e44cdba72399d494f541a30c8d not found: ID does not exist" Nov 03 21:26:36.965015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:36.964982 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:37.084975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:37.084946 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5"] Nov 03 21:26:37.086784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:37.086761 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c2e645a4-78bd896fd8-7jvj5"] Nov 03 21:26:37.694887 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:37.694862 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:37.952532 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:37.952428 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:38.202576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:38.202497 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:38.438347 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:38.438318 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:38.505649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:38.505562 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea3cca1-2dd6-453d-9b70-5092fc85beb5" path="/var/lib/kubelet/pods/7ea3cca1-2dd6-453d-9b70-5092fc85beb5/volumes" Nov 03 21:26:38.670714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:38.670674 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:38.902163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:38.902135 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:39.262832 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:39.262763 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:39.491024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:39.490996 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:39.721133 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:39.721102 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:40.084643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.084532 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk"] Nov 03 21:26:40.085062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085043 5931 cpu_manager.go:414] "containerMap: removing container" podUID="d10650a0-df9f-45f3-b49f-60b0ba5eb47c" containerName="payload-processing" Nov 03 21:26:40.085146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085065 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10650a0-df9f-45f3-b49f-60b0ba5eb47c" containerName="payload-processing" Nov 03 21:26:40.085146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085077 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7ea3cca1-2dd6-453d-9b70-5092fc85beb5" containerName="maas-api" Nov 03 21:26:40.085146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085083 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea3cca1-2dd6-453d-9b70-5092fc85beb5" containerName="maas-api" Nov 03 21:26:40.085146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085103 5931 cpu_manager.go:414] "containerMap: removing container" podUID="755b28c1-a3d8-41c4-8241-ebd9c853dcd0" containerName="maas-api" Nov 03 21:26:40.085146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085109 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755b28c1-a3d8-41c4-8241-ebd9c853dcd0" containerName="maas-api" Nov 03 21:26:40.085396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085255 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10650a0-df9f-45f3-b49f-60b0ba5eb47c" containerName="payload-processing" Nov 03 21:26:40.085396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085266 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea3cca1-2dd6-453d-9b70-5092fc85beb5" containerName="maas-api" Nov 03 21:26:40.085396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.085278 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="755b28c1-a3d8-41c4-8241-ebd9c853dcd0" containerName="maas-api" Nov 03 21:26:40.087930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.087910 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.147815 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.147787 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk"] Nov 03 21:26:40.185799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.185777 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.185905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.185814 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9b7e36-937b-475b-a231-c17b97becd85-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.185905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.185857 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.185980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.185948 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.186014 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.185985 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9b7e36-937b-475b-a231-c17b97becd85-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.186082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.186066 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpcq\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-kube-api-access-crpcq\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.186116 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.186097 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.186150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.186122 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.186206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.186194 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.196715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.196695 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7ee809-openshift-default-dockercfg-qsllw\"" Nov 03 21:26:40.270591 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.270550 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:40.286823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.286796 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpcq\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-kube-api-access-crpcq\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.286960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.286835 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.286960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.286869 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.286960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.286901 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.286958 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.286996 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9b7e36-937b-475b-a231-c17b97becd85-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287044 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287080 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287105 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9b7e36-937b-475b-a231-c17b97becd85-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287257 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287386 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287440 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287524 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287500 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.287797 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.287775 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9b7e36-937b-475b-a231-c17b97becd85-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.289243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.289222 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.289594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.289576 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9b7e36-937b-475b-a231-c17b97becd85-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.343938 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.343904 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.344145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.344126 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpcq\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-kube-api-access-crpcq\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.399142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.399117 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:40.525650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.525603 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:40.824257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.824232 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk"] Nov 03 21:26:40.826465 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:26:40.826437 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9b7e36_937b_475b_a231_c17b97becd85.slice/crio-b8517a44d58e3d9429e68a63881075d112084de74fce254531966ff5b11fc1e1 WatchSource:0}: Error finding container b8517a44d58e3d9429e68a63881075d112084de74fce254531966ff5b11fc1e1: Status 404 returned error can't find the container with id b8517a44d58e3d9429e68a63881075d112084de74fce254531966ff5b11fc1e1 Nov 03 21:26:40.828590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.828563 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:40.828702 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.828645 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:40.828702 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.828674 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:26:40.890523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.890497 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" event={"ID":"5a9b7e36-937b-475b-a231-c17b97becd85","Type":"ContainerStarted","Data":"1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3"} Nov 03 21:26:40.890606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:40.890559 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" event={"ID":"5a9b7e36-937b-475b-a231-c17b97becd85","Type":"ContainerStarted","Data":"b8517a44d58e3d9429e68a63881075d112084de74fce254531966ff5b11fc1e1"} Nov 03 21:26:41.007346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:41.007297 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podStartSLOduration=1.007283845 podStartE2EDuration="1.007283845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:41.004212047 +0000 UTC m=+668.615059866" watchObservedRunningTime="2026-11-03 21:26:41.007283845 +0000 UTC m=+668.618131688" Nov 03 21:26:41.399918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:41.399884 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:41.404963 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:41.404939 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:41.895576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:41.895553 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:41.896539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:41.896518 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:26:41.968079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:41.968055 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9"] Nov 03 21:26:42.110191 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.110168 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:26:42.203600 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203525 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-envoy\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.203740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203632 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-istio-token\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-istio-token\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.203740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203680 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-data\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.203740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203729 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d74c17-7790-4833-8cfa-06fcbec02964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d74c17-7790-4833-8cfa-06fcbec02964-istiod-ca-cert\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.203905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203813 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-socket\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.203905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203850 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-kube-api-access-9ggkf\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-kube-api-access-9ggkf\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.203905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203880 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37d74c17-7790-4833-8cfa-06fcbec02964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d74c17-7790-4833-8cfa-06fcbec02964-istio-podinfo\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.204050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203911 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-credential-socket\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.204050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.203948 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-data" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:42.204143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.204070 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-socket" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:42.204143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.204110 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d74c17-7790-4833-8cfa-06fcbec02964-istiod-ca-cert" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:42.204358 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.204318 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-credential-socket" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:42.204445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.204367 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.204445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.204391 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.204445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.204410 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d74c17-7790-4833-8cfa-06fcbec02964-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.206066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.206043 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37d74c17-7790-4833-8cfa-06fcbec02964-istio-podinfo" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:26:42.206171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.206073 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-kube-api-access-9ggkf" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "kube-api-access-9ggkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:42.206223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.206208 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-envoy" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:42.206223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.206208 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-istio-token" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:42.305354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.305325 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-certs\") pod \"37d74c17-7790-4833-8cfa-06fcbec02964\" (UID: \"37d74c17-7790-4833-8cfa-06fcbec02964\") " Nov 03 21:26:42.305568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.305557 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.305630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.305570 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggkf\" (UniqueName: \"kubernetes.io/projected/37d74c17-7790-4833-8cfa-06fcbec02964-kube-api-access-9ggkf\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.305630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.305580 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d74c17-7790-4833-8cfa-06fcbec02964-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.305630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.305590 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.305630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.305599 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.305630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.305580 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-certs" pod "37d74c17-7790-4833-8cfa-06fcbec02964" (UID: "37d74c17-7790-4833-8cfa-06fcbec02964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:42.396051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.396020 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:42.406472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.406448 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d74c17-7790-4833-8cfa-06fcbec02964-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:42.635965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.635908 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:42.890561 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.890532 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:42.901280 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.901256 5931 generic.go:356] "Generic (PLEG): container finished" podID="37d74c17-7790-4833-8cfa-06fcbec02964" containerID="0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9" exitCode=0 Nov 03 21:26:42.901389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.901311 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" Nov 03 21:26:42.901389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.901338 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" event={"ID":"37d74c17-7790-4833-8cfa-06fcbec02964","Type":"ContainerDied","Data":"0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9"} Nov 03 21:26:42.901459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.901395 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7" event={"ID":"37d74c17-7790-4833-8cfa-06fcbec02964","Type":"ContainerDied","Data":"7462ad65e164b06cda74c3369056271a879ceabfdd43500bc712d40a65f19c42"} Nov 03 21:26:42.901459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.901416 5931 scope.go:122] "RemoveContainer" containerID="0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9" Nov 03 21:26:42.910476 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.910459 5931 scope.go:122] "RemoveContainer" containerID="d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349" Nov 03 21:26:42.918701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.918684 5931 scope.go:122] "RemoveContainer" containerID="0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9" Nov 03 21:26:42.918986 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:42.918969 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9\": container with ID starting with 0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9 not found: ID does not exist" containerID="0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9" Nov 03 21:26:42.919049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.918993 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9"} err="failed to get container status \"0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9\": rpc error: code = NotFound desc = could not find container \"0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9\": container with ID starting with 0b6b5b8c4ca008c27530894978f27cdc195cb3473abf3e86f281ffbe7c9df2c9 not found: ID does not exist" Nov 03 21:26:42.919049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.919010 5931 scope.go:122] "RemoveContainer" containerID="d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349" Nov 03 21:26:42.919231 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:42.919216 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349\": container with ID starting with d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349 not found: ID does not exist" containerID="d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349" Nov 03 21:26:42.919270 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:42.919236 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349"} err="failed to get container status \"d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349\": rpc error: code = NotFound desc = could not find container \"d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349\": container with ID starting with d985df5cb74414c2f863f3a7b51919bf0a128b164814017c60b44939e8338349 not found: ID does not exist" Nov 03 21:26:43.066599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:43.066568 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7"] Nov 03 21:26:43.069905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:43.069883 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-889495a7-openshift-default-d4df596ff-h64h7"] Nov 03 21:26:44.020225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:44.020198 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:44.244198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:44.244164 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:44.477022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:44.476988 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:44.507105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:44.507076 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" path="/var/lib/kubelet/pods/37d74c17-7790-4833-8cfa-06fcbec02964/volumes" Nov 03 21:26:44.749100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:44.749023 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:45.509897 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:45.509852 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:46.045821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.045789 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:46.338964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.338889 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:46.576264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.576212 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:46.804158 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.804129 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:46.885234 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.885205 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w"] Nov 03 21:26:46.885605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.885594 5931 cpu_manager.go:414] "containerMap: removing container" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" Nov 03 21:26:46.885605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.885618 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" Nov 03 21:26:46.885728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.885656 5931 cpu_manager.go:414] "containerMap: removing container" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" Nov 03 21:26:46.885728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.885661 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" Nov 03 21:26:46.885728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.885719 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" Nov 03 21:26:46.885814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.885729 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d74c17-7790-4833-8cfa-06fcbec02964" containerName="istio-proxy" Nov 03 21:26:46.887903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.887885 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:46.941859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.941823 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549c57f9-0b18-48da-9712-73248d9d5cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:46.941973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.941881 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsnwn\" (UniqueName: \"kubernetes.io/projected/549c57f9-0b18-48da-9712-73248d9d5cb8-kube-api-access-jsnwn\") pod \"payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:46.968840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.968807 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" podUID="1bec2544-7d64-4ecd-b486-6ddd16c01914" containerName="payload-processing" containerID="cri-o://75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951" gracePeriod=25 Nov 03 21:26:46.988910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.988882 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d71a74\"" Nov 03 21:26:46.989167 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.989120 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d71a74-dockercfg-2jbwf\"" Nov 03 21:26:46.991782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:46.991757 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w"] Nov 03 21:26:47.043199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.043178 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549c57f9-0b18-48da-9712-73248d9d5cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:47.043295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.043213 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsnwn\" (UniqueName: \"kubernetes.io/projected/549c57f9-0b18-48da-9712-73248d9d5cb8-kube-api-access-jsnwn\") pod \"payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:47.043870 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.043851 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549c57f9-0b18-48da-9712-73248d9d5cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:47.099877 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.099849 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsnwn\" (UniqueName: \"kubernetes.io/projected/549c57f9-0b18-48da-9712-73248d9d5cb8-kube-api-access-jsnwn\") pod \"payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:47.199266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.199241 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:47.219914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.219892 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:26:47.246053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.245273 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bec2544-7d64-4ecd-b486-6ddd16c01914-kube-api-access-tkpb4\" (UniqueName: \"kubernetes.io/projected/1bec2544-7d64-4ecd-b486-6ddd16c01914-kube-api-access-tkpb4\") pod \"1bec2544-7d64-4ecd-b486-6ddd16c01914\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " Nov 03 21:26:47.246053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.245355 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bec2544-7d64-4ecd-b486-6ddd16c01914-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec2544-7d64-4ecd-b486-6ddd16c01914-plugins-config-volume\") pod \"1bec2544-7d64-4ecd-b486-6ddd16c01914\" (UID: \"1bec2544-7d64-4ecd-b486-6ddd16c01914\") " Nov 03 21:26:47.246053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.245962 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bec2544-7d64-4ecd-b486-6ddd16c01914-plugins-config-volume" pod "1bec2544-7d64-4ecd-b486-6ddd16c01914" (UID: "1bec2544-7d64-4ecd-b486-6ddd16c01914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:47.248178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.248111 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bec2544-7d64-4ecd-b486-6ddd16c01914-kube-api-access-tkpb4" pod "1bec2544-7d64-4ecd-b486-6ddd16c01914" (UID: "1bec2544-7d64-4ecd-b486-6ddd16c01914"). InnerVolumeSpecName "kube-api-access-tkpb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:47.346346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.346318 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec2544-7d64-4ecd-b486-6ddd16c01914-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:47.346346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.346345 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkpb4\" (UniqueName: \"kubernetes.io/projected/1bec2544-7d64-4ecd-b486-6ddd16c01914-kube-api-access-tkpb4\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:47.567623 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.567572 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w"] Nov 03 21:26:47.570899 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:26:47.570864 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549c57f9_0b18_48da_9712_73248d9d5cb8.slice/crio-99bd743a925e525157f4364462f69f3e3829fe198834242262278a4bfae2adb5 WatchSource:0}: Error finding container 99bd743a925e525157f4364462f69f3e3829fe198834242262278a4bfae2adb5: Status 404 returned error can't find the container with id 99bd743a925e525157f4364462f69f3e3829fe198834242262278a4bfae2adb5 Nov 03 21:26:47.933177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.933090 5931 generic.go:356] "Generic (PLEG): container finished" podID="1bec2544-7d64-4ecd-b486-6ddd16c01914" containerID="75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951" exitCode=0 Nov 03 21:26:47.933177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.933138 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" Nov 03 21:26:47.933748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.933178 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" event={"ID":"1bec2544-7d64-4ecd-b486-6ddd16c01914","Type":"ContainerDied","Data":"75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951"} Nov 03 21:26:47.933748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.933229 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9" event={"ID":"1bec2544-7d64-4ecd-b486-6ddd16c01914","Type":"ContainerDied","Data":"f12228014913127f6888722ce10f88eac2a3e11622415ac71f00d0295b2fdcb8"} Nov 03 21:26:47.933748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.933250 5931 scope.go:122] "RemoveContainer" containerID="75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951" Nov 03 21:26:47.935041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.935016 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" event={"ID":"549c57f9-0b18-48da-9712-73248d9d5cb8","Type":"ContainerStarted","Data":"b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e"} Nov 03 21:26:47.935135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.935077 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" event={"ID":"549c57f9-0b18-48da-9712-73248d9d5cb8","Type":"ContainerStarted","Data":"99bd743a925e525157f4364462f69f3e3829fe198834242262278a4bfae2adb5"} Nov 03 21:26:47.935135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.935112 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:47.942964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.942948 5931 scope.go:122] "RemoveContainer" containerID="75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951" Nov 03 21:26:47.943189 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:47.943168 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951\": container with ID starting with 75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951 not found: ID does not exist" containerID="75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951" Nov 03 21:26:47.943244 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:47.943196 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951"} err="failed to get container status \"75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951\": rpc error: code = NotFound desc = could not find container \"75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951\": container with ID starting with 75952743cbc75dbf86c5696177beb9c1c0b86dbfbd800723457718db06c21951 not found: ID does not exist" Nov 03 21:26:48.097978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:48.097936 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" podStartSLOduration=2.097922768 podStartE2EDuration="2.097922768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:48.096935505 +0000 UTC m=+675.707783319" watchObservedRunningTime="2026-11-03 21:26:48.097922768 +0000 UTC m=+675.708770585" Nov 03 21:26:48.208823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:48.208145 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9"] Nov 03 21:26:48.209947 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:48.209920 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35257bc0-7f58878f84-kh9f9"] Nov 03 21:26:48.504666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:48.504563 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bec2544-7d64-4ecd-b486-6ddd16c01914" path="/var/lib/kubelet/pods/1bec2544-7d64-4ecd-b486-6ddd16c01914/volumes" Nov 03 21:26:49.684177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:49.684142 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:50.394360 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:50.394319 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:50.624077 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:50.624035 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:51.136779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:51.136748 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:51.386437 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:51.386408 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:53.921320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:53.921287 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv"] Nov 03 21:26:53.930305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:53.929398 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" podUID="1bbc622c-fcfb-41f5-a09d-999d4ade61e1" containerName="maas-api" containerID="cri-o://16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3" gracePeriod=30 Nov 03 21:26:53.930936 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:53.930895 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:54.151368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.151334 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:54.181294 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.181244 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:26:54.205745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.205725 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-kube-api-access-7c2j6\" (UniqueName: \"kubernetes.io/projected/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-kube-api-access-7c2j6\") pod \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " Nov 03 21:26:54.205846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.205777 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls\") pod \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " Nov 03 21:26:54.205846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.205825 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-openshift-service-ca\") pod \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " Nov 03 21:26:54.205923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.205846 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-metrics-certs\") pod \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\" (UID: \"1bbc622c-fcfb-41f5-a09d-999d4ade61e1\") " Nov 03 21:26:54.206171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.206149 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-openshift-service-ca" pod "1bbc622c-fcfb-41f5-a09d-999d4ade61e1" (UID: "1bbc622c-fcfb-41f5-a09d-999d4ade61e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:54.206267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.206253 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:54.207835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.207812 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-kube-api-access-7c2j6" pod "1bbc622c-fcfb-41f5-a09d-999d4ade61e1" (UID: "1bbc622c-fcfb-41f5-a09d-999d4ade61e1"). InnerVolumeSpecName "kube-api-access-7c2j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:54.207917 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.207883 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls" pod "1bbc622c-fcfb-41f5-a09d-999d4ade61e1" (UID: "1bbc622c-fcfb-41f5-a09d-999d4ade61e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:54.207994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.207980 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-metrics-certs" pod "1bbc622c-fcfb-41f5-a09d-999d4ade61e1" (UID: "1bbc622c-fcfb-41f5-a09d-999d4ade61e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:54.234474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.234450 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd"] Nov 03 21:26:54.234853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.234839 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1bbc622c-fcfb-41f5-a09d-999d4ade61e1" containerName="maas-api" Nov 03 21:26:54.234893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.234855 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbc622c-fcfb-41f5-a09d-999d4ade61e1" containerName="maas-api" Nov 03 21:26:54.234929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.234895 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1bec2544-7d64-4ecd-b486-6ddd16c01914" containerName="payload-processing" Nov 03 21:26:54.234929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.234900 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bec2544-7d64-4ecd-b486-6ddd16c01914" containerName="payload-processing" Nov 03 21:26:54.234990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.234960 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bec2544-7d64-4ecd-b486-6ddd16c01914" containerName="payload-processing" Nov 03 21:26:54.234990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.234967 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbc622c-fcfb-41f5-a09d-999d4ade61e1" containerName="maas-api" Nov 03 21:26:54.236808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.236793 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.288861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.288842 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7ee809\"" Nov 03 21:26:54.289009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.288996 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7ee809-dockercfg-sgmwf\"" Nov 03 21:26:54.296484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.296464 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd"] Nov 03 21:26:54.312649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.307700 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqr8f\" (UniqueName: \"kubernetes.io/projected/6f745fb4-1903-4e95-b3c9-92429f5400d5-kube-api-access-sqr8f\") pod \"payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.312649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.307846 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f745fb4-1903-4e95-b3c9-92429f5400d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.312649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.307916 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:54.312649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.307932 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2j6\" (UniqueName: \"kubernetes.io/projected/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-kube-api-access-7c2j6\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:54.312649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.307948 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbc622c-fcfb-41f5-a09d-999d4ade61e1-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:26:54.384426 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.384389 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh"] Nov 03 21:26:54.386740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.386717 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.409535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.409421 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl6dh\" (UniqueName: \"kubernetes.io/projected/2a656523-f835-437b-80d2-fff51c87f061-kube-api-access-kl6dh\") pod \"payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.410466 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.409545 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqr8f\" (UniqueName: \"kubernetes.io/projected/6f745fb4-1903-4e95-b3c9-92429f5400d5-kube-api-access-sqr8f\") pod \"payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.410466 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.410092 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a656523-f835-437b-80d2-fff51c87f061-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.410466 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.410196 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f745fb4-1903-4e95-b3c9-92429f5400d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.411237 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.411119 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f745fb4-1903-4e95-b3c9-92429f5400d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.445499 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.445433 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh"] Nov 03 21:26:54.465993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.465964 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqr8f\" (UniqueName: \"kubernetes.io/projected/6f745fb4-1903-4e95-b3c9-92429f5400d5-kube-api-access-sqr8f\") pod \"payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.511097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.511018 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a656523-f835-437b-80d2-fff51c87f061-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.511161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.511111 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6dh\" (UniqueName: \"kubernetes.io/projected/2a656523-f835-437b-80d2-fff51c87f061-kube-api-access-kl6dh\") pod \"payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.511749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.511731 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a656523-f835-437b-80d2-fff51c87f061-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.549235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.549208 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:54.568244 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.568229 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl6dh\" (UniqueName: \"kubernetes.io/projected/2a656523-f835-437b-80d2-fff51c87f061-kube-api-access-kl6dh\") pod \"payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.698586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.698517 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:54.916525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.916429 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd"] Nov 03 21:26:54.919144 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:26:54.919118 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f745fb4_1903_4e95_b3c9_92429f5400d5.slice/crio-37b896a07c554dcab1d4c373d61fe78794a5e9a92a0831a400c02fc4aa27fb9f WatchSource:0}: Error finding container 37b896a07c554dcab1d4c373d61fe78794a5e9a92a0831a400c02fc4aa27fb9f: Status 404 returned error can't find the container with id 37b896a07c554dcab1d4c373d61fe78794a5e9a92a0831a400c02fc4aa27fb9f Nov 03 21:26:54.969785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.969758 5931 generic.go:356] "Generic (PLEG): container finished" podID="1bbc622c-fcfb-41f5-a09d-999d4ade61e1" containerID="16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3" exitCode=0 Nov 03 21:26:54.970101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.969800 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" Nov 03 21:26:54.970101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.969836 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" event={"ID":"1bbc622c-fcfb-41f5-a09d-999d4ade61e1","Type":"ContainerDied","Data":"16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3"} Nov 03 21:26:54.970101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.969884 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv" event={"ID":"1bbc622c-fcfb-41f5-a09d-999d4ade61e1","Type":"ContainerDied","Data":"2bdc3cab6f685c44d3510eb7f3cb97fd7dfed6cdf54555f6c4b3a24bf3d40fa9"} Nov 03 21:26:54.970101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.969908 5931 scope.go:122] "RemoveContainer" containerID="16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3" Nov 03 21:26:54.971108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.971087 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" event={"ID":"6f745fb4-1903-4e95-b3c9-92429f5400d5","Type":"ContainerStarted","Data":"37b896a07c554dcab1d4c373d61fe78794a5e9a92a0831a400c02fc4aa27fb9f"} Nov 03 21:26:54.980001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.979981 5931 scope.go:122] "RemoveContainer" containerID="16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3" Nov 03 21:26:54.980243 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:54.980221 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3\": container with ID starting with 16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3 not found: ID does not exist" containerID="16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3" Nov 03 21:26:54.980320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:54.980251 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3"} err="failed to get container status \"16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3\": rpc error: code = NotFound desc = could not find container \"16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3\": container with ID starting with 16762c81ea2c22eede7e5cb4d69b7c0b2069aa41cf12892ce642b57b730c71f3 not found: ID does not exist" Nov 03 21:26:55.066518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.066401 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh"] Nov 03 21:26:55.068661 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:26:55.068638 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a656523_f835_437b_80d2_fff51c87f061.slice/crio-852292dfab586a264aeb4212b758ab2a223f8809b38bc32adf369966b79f48e3 WatchSource:0}: Error finding container 852292dfab586a264aeb4212b758ab2a223f8809b38bc32adf369966b79f48e3: Status 404 returned error can't find the container with id 852292dfab586a264aeb4212b758ab2a223f8809b38bc32adf369966b79f48e3 Nov 03 21:26:55.131097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.131077 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv"] Nov 03 21:26:55.137491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.137472 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35257bc0-6496fd777f-6m2bv"] Nov 03 21:26:55.494627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.494582 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct"] Nov 03 21:26:55.496845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.496826 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.522656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.522606 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nz2\" (UniqueName: \"kubernetes.io/projected/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-kube-api-access-p7nz2\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.522756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.522717 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.522801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.522787 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.522890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.522876 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-metrics-certs\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.548826 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.548808 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:26:55.548902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.548823 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-d71a74-serving-cert\"" Nov 03 21:26:55.548965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.548952 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nbmd6\"" Nov 03 21:26:55.549037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.549021 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:26:55.549082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.549042 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:26:55.555405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.555384 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct"] Nov 03 21:26:55.624152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.624129 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.624243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.624180 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-metrics-certs\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.624243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.624221 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nz2\" (UniqueName: \"kubernetes.io/projected/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-kube-api-access-p7nz2\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.624371 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:55.624262 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-serving-cert: secret "maas-api-e2e-worker-w2-d71a74-serving-cert" not found Nov 03 21:26:55.624371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.624273 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.624371 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:26:55.624320 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls podName:f95cb44f-4d6a-48d6-a0a2-6e912d5ff997 nodeName:}" failed. No retries permitted until 2026-11-03 21:26:56.124301619 +0000 UTC m=+683.735149429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls") pod "maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" (UID: "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997") : secret "maas-api-e2e-worker-w2-d71a74-serving-cert" not found Nov 03 21:26:55.624904 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.624884 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.626686 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.626670 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-metrics-certs\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.680632 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.680586 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nz2\" (UniqueName: \"kubernetes.io/projected/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-kube-api-access-p7nz2\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:55.783451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.783361 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:55.984853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.984809 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" event={"ID":"6f745fb4-1903-4e95-b3c9-92429f5400d5","Type":"ContainerStarted","Data":"00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a"} Nov 03 21:26:55.985303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.984893 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:26:55.986700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.986676 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" event={"ID":"2a656523-f835-437b-80d2-fff51c87f061","Type":"ContainerStarted","Data":"ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58"} Nov 03 21:26:55.986854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.986728 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" event={"ID":"2a656523-f835-437b-80d2-fff51c87f061","Type":"ContainerStarted","Data":"852292dfab586a264aeb4212b758ab2a223f8809b38bc32adf369966b79f48e3"} Nov 03 21:26:55.986854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:55.986759 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:26:56.009260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.009231 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:56.099326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.099285 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" podStartSLOduration=2.09927434 podStartE2EDuration="2.09927434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:56.097004741 +0000 UTC m=+683.707852556" watchObservedRunningTime="2026-11-03 21:26:56.09927434 +0000 UTC m=+683.710122210" Nov 03 21:26:56.128975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.128955 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:56.131307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.131283 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:56.211196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.211147 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" podStartSLOduration=2.211130744 podStartE2EDuration="2.211130744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:56.208893421 +0000 UTC m=+683.819741236" watchObservedRunningTime="2026-11-03 21:26:56.211130744 +0000 UTC m=+683.821978560" Nov 03 21:26:56.238373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.238342 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:56.407866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.407809 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:56.506602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.506572 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbc622c-fcfb-41f5-a09d-999d4ade61e1" path="/var/lib/kubelet/pods/1bbc622c-fcfb-41f5-a09d-999d4ade61e1/volumes" Nov 03 21:26:56.509451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.509419 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:56.732798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.732725 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:56.789316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.789285 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct"] Nov 03 21:26:56.792136 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:26:56.792110 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95cb44f_4d6a_48d6_a0a2_6e912d5ff997.slice/crio-a3d741da6350fe904451d70dcc6c64a572c7ac1f669a253b8d4482a30145ad44 WatchSource:0}: Error finding container a3d741da6350fe904451d70dcc6c64a572c7ac1f669a253b8d4482a30145ad44: Status 404 returned error can't find the container with id a3d741da6350fe904451d70dcc6c64a572c7ac1f669a253b8d4482a30145ad44 Nov 03 21:26:56.994365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:56.994283 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" event={"ID":"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997","Type":"ContainerStarted","Data":"a3d741da6350fe904451d70dcc6c64a572c7ac1f669a253b8d4482a30145ad44"} Nov 03 21:26:57.041482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:57.041453 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:57.310723 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:57.310640 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:58.000575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:58.000499 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" event={"ID":"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997","Type":"ContainerStarted","Data":"6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c"} Nov 03 21:26:58.000913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:58.000652 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:26:58.114649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:58.114580 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" podStartSLOduration=2.176391588 podStartE2EDuration="3.114567423s" totalImagesPullingTime="938.175835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:58.113157019 +0000 UTC m=+685.724004835" watchObservedRunningTime="2026-11-03 21:26:58.114567423 +0000 UTC m=+685.725415260" Nov 03 21:26:58.943571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:58.943546 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 21:26:59.193922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:26:59.193850 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:00.055106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.055077 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:00.628407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.628377 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss"] Nov 03 21:27:00.630927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.630909 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.673009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.672981 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-metrics-certs\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.673154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.673041 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.673199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.673149 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae84c73-222d-40e4-ba77-e046ef07b02f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.673235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.673218 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6bk\" (UniqueName: \"kubernetes.io/projected/6ae84c73-222d-40e4-ba77-e046ef07b02f-kube-api-access-rl6bk\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.686212 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.686187 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:00.692309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.692294 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7ee809-serving-cert\"" Nov 03 21:27:00.734162 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.734141 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss"] Nov 03 21:27:00.774075 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.774053 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.774204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.774113 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae84c73-222d-40e4-ba77-e046ef07b02f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.774204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.774189 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6bk\" (UniqueName: \"kubernetes.io/projected/6ae84c73-222d-40e4-ba77-e046ef07b02f-kube-api-access-rl6bk\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.774318 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:00.774210 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-serving-cert: secret "maas-api-e2e-worker-w0-7ee809-serving-cert" not found Nov 03 21:27:00.774318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.774237 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-metrics-certs\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.774318 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:00.774263 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls podName:6ae84c73-222d-40e4-ba77-e046ef07b02f nodeName:}" failed. No retries permitted until 2026-11-03 21:27:01.27424824 +0000 UTC m=+688.885096052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls") pod "maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" (UID: "6ae84c73-222d-40e4-ba77-e046ef07b02f") : secret "maas-api-e2e-worker-w0-7ee809-serving-cert" not found Nov 03 21:27:00.774836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.774819 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae84c73-222d-40e4-ba77-e046ef07b02f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.776689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.776669 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-metrics-certs\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.831204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.831185 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6bk\" (UniqueName: \"kubernetes.io/projected/6ae84c73-222d-40e4-ba77-e046ef07b02f-kube-api-access-rl6bk\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:00.934179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:00.934099 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:01.155603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:01.155569 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:01.280757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:01.280680 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:01.283226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:01.283205 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:01.384284 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:01.384259 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:01.545397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:01.545344 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:01.912351 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:01.912323 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss"] Nov 03 21:27:01.916717 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:27:01.916674 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae84c73_222d_40e4_ba77_e046ef07b02f.slice/crio-f64d977bc41c0cde17d0f2891f6cac8e3cab609c6431687a0bc08049acbbdc16 WatchSource:0}: Error finding container f64d977bc41c0cde17d0f2891f6cac8e3cab609c6431687a0bc08049acbbdc16: Status 404 returned error can't find the container with id f64d977bc41c0cde17d0f2891f6cac8e3cab609c6431687a0bc08049acbbdc16 Nov 03 21:27:02.023496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:02.023464 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" event={"ID":"6ae84c73-222d-40e4-ba77-e046ef07b02f","Type":"ContainerStarted","Data":"f64d977bc41c0cde17d0f2891f6cac8e3cab609c6431687a0bc08049acbbdc16"} Nov 03 21:27:03.029495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:03.029422 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" event={"ID":"6ae84c73-222d-40e4-ba77-e046ef07b02f","Type":"ContainerStarted","Data":"dd017ee18931c435a053135bd42cbcb57664d48cdf518d9864e97619dc872b3f"} Nov 03 21:27:03.029863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:03.029558 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:03.067648 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:03.067591 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:27:03.143477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:03.143431 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" podStartSLOduration=2.353508451 podStartE2EDuration="3.143417677s" totalImagesPullingTime="789.909226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:03.142020875 +0000 UTC m=+690.752868690" watchObservedRunningTime="2026-11-03 21:27:03.143417677 +0000 UTC m=+690.754265492" Nov 03 21:27:04.011369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:04.011343 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 21:27:04.767680 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:04.767651 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:05.016595 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:05.016562 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:05.372504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:05.372474 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:06.149138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:06.149103 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:06.997104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:06.997076 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 21:27:06.997281 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:06.997259 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 21:27:09.054083 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:09.054052 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 21:27:10.642596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.642550 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:10.689288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.689256 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4"] Nov 03 21:27:10.692987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.692963 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.746941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.746908 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-da5f0b-serving-cert\"" Nov 03 21:27:10.766839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.766811 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-metrics-certs\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.766970 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.766879 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.766970 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.766932 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdlr5\" (UniqueName: \"kubernetes.io/projected/443b17c9-5a0b-496f-a013-a55819000d43-kube-api-access-fdlr5\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.767041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.767012 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443b17c9-5a0b-496f-a013-a55819000d43-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.798550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.798523 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4"] Nov 03 21:27:10.868575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.868539 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-metrics-certs\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.868924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.868903 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.869000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.868956 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdlr5\" (UniqueName: \"kubernetes.io/projected/443b17c9-5a0b-496f-a013-a55819000d43-kube-api-access-fdlr5\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.869114 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:10.869012 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-serving-cert: secret "maas-api-e2e-worker-w3-da5f0b-serving-cert" not found Nov 03 21:27:10.869114 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:10.869079 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls podName:443b17c9-5a0b-496f-a013-a55819000d43 nodeName:}" failed. No retries permitted until 2026-11-03 21:27:11.369059109 +0000 UTC m=+698.979906926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls") pod "maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" (UID: "443b17c9-5a0b-496f-a013-a55819000d43") : secret "maas-api-e2e-worker-w3-da5f0b-serving-cert" not found Nov 03 21:27:10.869114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.869013 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443b17c9-5a0b-496f-a013-a55819000d43-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.869633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.869594 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443b17c9-5a0b-496f-a013-a55819000d43-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.871034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.871016 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-metrics-certs\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:10.872384 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.872362 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:10.948864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:10.948798 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdlr5\" (UniqueName: \"kubernetes.io/projected/443b17c9-5a0b-496f-a013-a55819000d43-kube-api-access-fdlr5\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:11.112806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:11.112770 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:11.346634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:11.344421 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:11.372448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:11.372425 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:11.374849 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:11.374831 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls\") pod \"maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:11.577619 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:11.577588 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:11.605553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:11.605499 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:12.006012 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:12.005988 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4"] Nov 03 21:27:12.008596 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:27:12.008567 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443b17c9_5a0b_496f_a013_a55819000d43.slice/crio-300b3f79091e382fd647fca130184846692f77c2f6161021a94a2b7da82ad86f WatchSource:0}: Error finding container 300b3f79091e382fd647fca130184846692f77c2f6161021a94a2b7da82ad86f: Status 404 returned error can't find the container with id 300b3f79091e382fd647fca130184846692f77c2f6161021a94a2b7da82ad86f Nov 03 21:27:12.080036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:12.080010 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" event={"ID":"443b17c9-5a0b-496f-a013-a55819000d43","Type":"ContainerStarted","Data":"300b3f79091e382fd647fca130184846692f77c2f6161021a94a2b7da82ad86f"} Nov 03 21:27:13.087802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:13.087775 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" event={"ID":"443b17c9-5a0b-496f-a013-a55819000d43","Type":"ContainerStarted","Data":"ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52"} Nov 03 21:27:13.088130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:13.087838 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:13.216043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:13.215976 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" podStartSLOduration=2.256127699 podStartE2EDuration="3.215960881s" totalImagesPullingTime="959.833182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:13.215737108 +0000 UTC m=+700.826584922" watchObservedRunningTime="2026-11-03 21:27:13.215960881 +0000 UTC m=+700.826808696" Nov 03 21:27:15.237802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:15.237750 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:16.747036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:16.746999 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:16.988310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:16.988273 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:17.248770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:17.248742 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:17.251101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:17.251080 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj"] Nov 03 21:27:17.251333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:17.251311 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" containerID="cri-o://43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada" gracePeriod=30 Nov 03 21:27:17.494102 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:17.494027 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:18.126591 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:18.126560 5931 generic.go:356] "Generic (PLEG): container finished" podID="5a9b7e36-937b-475b-a231-c17b97becd85" containerID="1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3" exitCode=0 Nov 03 21:27:18.127055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:18.126643 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" event={"ID":"5a9b7e36-937b-475b-a231-c17b97becd85","Type":"ContainerDied","Data":"1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3"} Nov 03 21:27:18.127055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:18.126985 5931 scope.go:122] "RemoveContainer" containerID="1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3" Nov 03 21:27:18.128060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:18.128025 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:27:18.128189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:18.128103 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:27:18.128189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:18.128149 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:27:18.189152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:18.189126 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:19.017910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:19.017881 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:19.098796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:19.098772 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 21:27:19.152388 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:19.152356 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" event={"ID":"5a9b7e36-937b-475b-a231-c17b97becd85","Type":"ContainerStarted","Data":"4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d"} Nov 03 21:27:19.254164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:19.254121 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:19.399939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:19.399907 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:27:19.400111 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:19.400084 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:27:19.400149 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:19.400124 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:27:20.400319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:20.400286 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:27:20.400746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:20.400337 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:27:21.400147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:21.400111 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:27:21.400326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:21.400163 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:27:22.013701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.013663 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:22.155136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.155107 5931 patch_prober.go:29] interesting pod/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:27:22.155293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.155157 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:27:22.399755 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.399723 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:27:22.399902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.399783 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:27:22.508927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.508904 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:27:22.599769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599743 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-socket\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.599908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599782 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-credential-socket\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.599908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599802 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-token\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-token\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.599908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599836 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-envoy\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.599908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599859 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-kube-api-access-2xqn2\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-kube-api-access-2xqn2\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.599908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599887 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-certs\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.600168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599914 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-data\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.600168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.599988 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istiod-ca-cert\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.600168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600020 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-socket" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:22.600168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600030 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-credential-socket" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:22.600168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600114 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-podinfo\") pod \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\" (UID: \"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973\") " Nov 03 21:27:22.600168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600116 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-certs" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:22.600387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600217 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-data" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:22.600387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600371 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istiod-ca-cert" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:22.600540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600521 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.600634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600541 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.600634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600555 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.600634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600568 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.600634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.600580 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.602268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.602241 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-envoy" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:22.602268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.602242 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-token" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:22.602568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.602548 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-kube-api-access-2xqn2" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "kube-api-access-2xqn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:22.602675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.602562 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-podinfo" pod "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" (UID: "6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:27:22.701315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.701287 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.701315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.701313 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.701490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.701322 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.701490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.701331 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqn2\" (UniqueName: \"kubernetes.io/projected/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973-kube-api-access-2xqn2\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:27:22.941691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:22.935968 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:23.182589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.182554 5931 generic.go:356] "Generic (PLEG): container finished" podID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerID="43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada" exitCode=0 Nov 03 21:27:23.183003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.182606 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" Nov 03 21:27:23.183003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.182652 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" event={"ID":"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973","Type":"ContainerDied","Data":"43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada"} Nov 03 21:27:23.183003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.182707 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj" event={"ID":"6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973","Type":"ContainerDied","Data":"9db35fe09df2b3be50073ee7b0f91ede381f3b8ac9d4d7926d1d913763d78ea5"} Nov 03 21:27:23.183003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.182730 5931 scope.go:122] "RemoveContainer" containerID="43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada" Nov 03 21:27:23.192375 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.192356 5931 scope.go:122] "RemoveContainer" containerID="8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b" Nov 03 21:27:23.201151 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.201136 5931 scope.go:122] "RemoveContainer" containerID="43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada" Nov 03 21:27:23.201404 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:23.201386 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada\": container with ID starting with 43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada not found: ID does not exist" containerID="43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada" Nov 03 21:27:23.201463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.201412 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada"} err="failed to get container status \"43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada\": rpc error: code = NotFound desc = could not find container \"43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada\": container with ID starting with 43b6a0d485ac92945534bdb221666b8874b56b1be7fea0fddd9a93d0808f9ada not found: ID does not exist" Nov 03 21:27:23.201463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.201432 5931 scope.go:122] "RemoveContainer" containerID="8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b" Nov 03 21:27:23.201697 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:23.201680 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b\": container with ID starting with 8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b not found: ID does not exist" containerID="8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b" Nov 03 21:27:23.201761 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.201702 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b"} err="failed to get container status \"8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b\": rpc error: code = NotFound desc = could not find container \"8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b\": container with ID starting with 8d4800f25a2c6dd5a2532803ba98bf11c30da99472d11871b4b2df986eb67b0b not found: ID does not exist" Nov 03 21:27:23.399565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.399532 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 21:27:23.399742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.399580 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 21:27:23.404289 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.404265 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj"] Nov 03 21:27:23.409710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.409690 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-daf21c29-openshift-default-bb896db69-5bwnj"] Nov 03 21:27:23.553439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.553373 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:23.803956 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:23.803841 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:24.050805 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:24.050772 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:24.403239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:24.403216 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:27:24.403714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:24.403644 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:27:24.404268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:24.404253 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:27:24.504278 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:24.504249 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" path="/var/lib/kubelet/pods/6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973/volumes" Nov 03 21:27:26.799329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.799297 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2"] Nov 03 21:27:26.799785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.799763 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" Nov 03 21:27:26.799785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.799786 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" Nov 03 21:27:26.799868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.799813 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" Nov 03 21:27:26.799868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.799820 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" Nov 03 21:27:26.799924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.799900 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" Nov 03 21:27:26.804569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.804553 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:26.857516 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.857494 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-6de8e6-serving-cert\"" Nov 03 21:27:26.909013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.908979 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2"] Nov 03 21:27:26.940557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.940528 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:26.940695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.940583 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtljb\" (UniqueName: \"kubernetes.io/projected/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-kube-api-access-dtljb\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:26.940767 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.940745 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:26.940817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:26.940796 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-metrics-certs\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.041381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.041352 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.041567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.041399 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtljb\" (UniqueName: \"kubernetes.io/projected/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-kube-api-access-dtljb\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.041567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.041474 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.041567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.041510 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-metrics-certs\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.041784 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:27.041683 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-serving-cert: secret "maas-api-e2e-worker-w4-6de8e6-serving-cert" not found Nov 03 21:27:27.041841 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:27.041778 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls podName:13d47dae-3cc7-4a35-87cb-fde3815aa4f0 nodeName:}" failed. No retries permitted until 2026-11-03 21:27:27.541756013 +0000 UTC m=+715.152603819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls") pod "maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" (UID: "13d47dae-3cc7-4a35-87cb-fde3815aa4f0") : secret "maas-api-e2e-worker-w4-6de8e6-serving-cert" not found Nov 03 21:27:27.042365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.042310 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.044148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.044127 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-metrics-certs\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.085998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.085932 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:27.101429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.101406 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtljb\" (UniqueName: \"kubernetes.io/projected/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-kube-api-access-dtljb\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.350414 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.350385 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:27.545660 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.545603 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.547970 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.547944 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.584925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.584898 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:27.716266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.716207 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:27.808337 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:27.808302 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:28.035313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:28.035231 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:28.104419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:28.104391 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2"] Nov 03 21:27:28.216787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:28.216748 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" event={"ID":"13d47dae-3cc7-4a35-87cb-fde3815aa4f0","Type":"ContainerStarted","Data":"432dcbe28215cda9a7078ac8b80ea1b0ad3470e558cd25e5dd2d80d294b69313"} Nov 03 21:27:29.223503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:29.223425 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" event={"ID":"13d47dae-3cc7-4a35-87cb-fde3815aa4f0","Type":"ContainerStarted","Data":"a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f"} Nov 03 21:27:29.223888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:29.223580 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:29.339362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:29.339313 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" podStartSLOduration=2.53619932 podStartE2EDuration="3.339298942s" totalImagesPullingTime="803.099622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:29.337650128 +0000 UTC m=+716.948497937" watchObservedRunningTime="2026-11-03 21:27:29.339298942 +0000 UTC m=+716.950146757" Nov 03 21:27:29.540645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:29.540533 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:30.197299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:30.197267 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:30.443029 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:30.442997 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:32.388415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:32.388379 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:33.066941 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:27:33.066906 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:27:35.235233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:35.235209 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 21:27:37.552714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:37.552686 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:39.640822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:39.640794 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:42.644382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:42.644312 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:46.507634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:46.507580 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:47.164401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:47.164371 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:47.422641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:47.422542 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:47.648891 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:47.648861 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:52.219307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:52.219272 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:52.443302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:52.443245 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:52.676720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:52.676689 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:54.929016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:54.928983 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:57.043119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:57.043086 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:58.439495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:27:58.439454 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:01.779808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:01.779776 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:03.067669 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:28:03.067604 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:28:05.725707 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:05.725671 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:06.370820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:06.370790 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:06.612666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:06.612633 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:06.962845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:06.962815 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:11.684923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:11.684890 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:16.331906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:16.331868 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:21.040604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:21.040564 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:25.837367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:25.837339 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:27.704773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:27.704743 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:29.036676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:29.036598 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:29.727500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:29.727467 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:29.963758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:29.963727 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:30.840737 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:30.840705 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:31.214974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:31.214880 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:33.066756 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:28:33.066722 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:28:35.643207 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:35.643167 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:39.991955 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:39.991917 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:40.386216 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:40.386185 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:45.113238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:45.113200 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:48.798629 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.798583 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq"] Nov 03 21:28:48.799081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.799069 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9aa6f1-9f36-4ea5-950d-8e79e6e8a973" containerName="istio-proxy" Nov 03 21:28:48.802376 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.802361 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:48.874829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.874806 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"kube-root-ca.crt\"" Nov 03 21:28:48.904706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.904680 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"openshift-service-ca.crt\"" Nov 03 21:28:48.904830 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.904820 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"premium-w0-7ee809-kserve-self-signed-certs\"" Nov 03 21:28:48.904941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.904923 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"default-dockercfg-qfhdp\"" Nov 03 21:28:48.906868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.906851 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq"] Nov 03 21:28:48.931855 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.931837 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005066fa-ff22-419b-af08-6f93eb95ce5c-tls-certs\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:48.931953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.931861 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd8nz\" (UniqueName: \"kubernetes.io/projected/005066fa-ff22-419b-af08-6f93eb95ce5c-kube-api-access-bd8nz\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:48.931953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.931891 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-dshm\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:48.931953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.931917 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-home\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:48.932090 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.932065 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-tmp-dir\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:48.932138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.932126 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-model-cache\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:48.978460 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:48.978427 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:49.032806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.032781 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005066fa-ff22-419b-af08-6f93eb95ce5c-tls-certs\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.032929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.032811 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd8nz\" (UniqueName: \"kubernetes.io/projected/005066fa-ff22-419b-af08-6f93eb95ce5c-kube-api-access-bd8nz\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.032929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.032863 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-dshm\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.032929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.032913 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-home\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.033092 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.032984 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-tmp-dir\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.033092 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.033051 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-model-cache\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.033372 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.033350 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-home\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.033458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.033394 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-model-cache\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.033458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.033434 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-tmp-dir\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.035283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.035265 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-dshm\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.035456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.035441 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005066fa-ff22-419b-af08-6f93eb95ce5c-tls-certs\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.088674 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.088624 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd8nz\" (UniqueName: \"kubernetes.io/projected/005066fa-ff22-419b-af08-6f93eb95ce5c-kube-api-access-bd8nz\") pod \"premium-w0-7ee809-kserve-5455d5c48d-6vvqq\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.113736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.113712 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:49.487288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.487251 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq"] Nov 03 21:28:49.489917 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:28:49.489896 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod005066fa_ff22_419b_af08_6f93eb95ce5c.slice/crio-e7eb432f7940438ad88ed8b2a75642fcc2d141aea1c2094172212aa7fe336f49 WatchSource:0}: Error finding container e7eb432f7940438ad88ed8b2a75642fcc2d141aea1c2094172212aa7fe336f49: Status 404 returned error can't find the container with id e7eb432f7940438ad88ed8b2a75642fcc2d141aea1c2094172212aa7fe336f49 Nov 03 21:28:49.647058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.647023 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" event={"ID":"005066fa-ff22-419b-af08-6f93eb95ce5c","Type":"ContainerStarted","Data":"b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08"} Nov 03 21:28:49.647184 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.647067 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" event={"ID":"005066fa-ff22-419b-af08-6f93eb95ce5c","Type":"ContainerStarted","Data":"e7eb432f7940438ad88ed8b2a75642fcc2d141aea1c2094172212aa7fe336f49"} Nov 03 21:28:49.760977 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.760897 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" podStartSLOduration=1.760884151 podStartE2EDuration="1.760884151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:49.758537425 +0000 UTC m=+797.369385239" watchObservedRunningTime="2026-11-03 21:28:49.760884151 +0000 UTC m=+797.371731965" Nov 03 21:28:49.826910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:49.826881 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:50.114228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:50.114200 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:54.533415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:54.533362 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:59.113873 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:59.113843 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:59.126138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:59.126114 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:28:59.227603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:59.227570 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:59.710199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:28:59.710173 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:29:03.067375 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:29:03.067333 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:29:03.707402 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.707374 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd"] Nov 03 21:29:03.711172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.711152 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.763995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.763966 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd"] Nov 03 21:29:03.765225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.765201 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-dshm\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.765381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.765245 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-model-cache\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.765381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.765272 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjxb\" (UniqueName: \"kubernetes.io/projected/c74eaac2-2350-4ec4-8601-a209a18500ef-kube-api-access-dmjxb\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.765491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.765422 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74eaac2-2350-4ec4-8601-a209a18500ef-tls-certs\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.765491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.765464 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-tmp-dir\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.765638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.765535 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-home\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.811023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.810998 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"openshift-service-ca.crt\"" Nov 03 21:29:03.811212 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.811045 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"kube-root-ca.crt\"" Nov 03 21:29:03.811212 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.811084 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"default-dockercfg-cmzvh\"" Nov 03 21:29:03.813079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.813061 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"premium-w3-da5f0b-kserve-self-signed-certs\"" Nov 03 21:29:03.866146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866121 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74eaac2-2350-4ec4-8601-a209a18500ef-tls-certs\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866157 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-tmp-dir\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866183 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-home\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866213 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-dshm\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866360 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866262 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-model-cache\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866360 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866298 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjxb\" (UniqueName: \"kubernetes.io/projected/c74eaac2-2350-4ec4-8601-a209a18500ef-kube-api-access-dmjxb\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866663 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866633 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-tmp-dir\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866670 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-home\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.866748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.866700 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-model-cache\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.868606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.868582 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-dshm\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.868741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.868725 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74eaac2-2350-4ec4-8601-a209a18500ef-tls-certs\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.922568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.922533 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjxb\" (UniqueName: \"kubernetes.io/projected/c74eaac2-2350-4ec4-8601-a209a18500ef-kube-api-access-dmjxb\") pod \"premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:03.925048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:03.925024 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:04.024658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:04.024575 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:04.167499 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:04.167466 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:04.408625 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:04.408221 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd"] Nov 03 21:29:04.426355 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:29:04.426318 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc74eaac2_2350_4ec4_8601_a209a18500ef.slice/crio-5f2fa578cd91333c7e0efb1eddc7bcf5c163812965e8b0404bd169006022fac6 WatchSource:0}: Error finding container 5f2fa578cd91333c7e0efb1eddc7bcf5c163812965e8b0404bd169006022fac6: Status 404 returned error can't find the container with id 5f2fa578cd91333c7e0efb1eddc7bcf5c163812965e8b0404bd169006022fac6 Nov 03 21:29:04.729898 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:04.729803 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" event={"ID":"c74eaac2-2350-4ec4-8601-a209a18500ef","Type":"ContainerStarted","Data":"5261b4c25850569605464806bc31688e98acc558a9370a5ea453b9a8cfc519ac"} Nov 03 21:29:04.729898 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:04.729851 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" event={"ID":"c74eaac2-2350-4ec4-8601-a209a18500ef","Type":"ContainerStarted","Data":"5f2fa578cd91333c7e0efb1eddc7bcf5c163812965e8b0404bd169006022fac6"} Nov 03 21:29:04.844211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:04.844156 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podStartSLOduration=1.8441427479999999 podStartE2EDuration="1.844142748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:04.84242192 +0000 UTC m=+812.453269734" watchObservedRunningTime="2026-11-03 21:29:04.844142748 +0000 UTC m=+812.454990737" Nov 03 21:29:05.025598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:05.025519 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:08.668417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:08.668378 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:11.593873 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:11.593780 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:13.335956 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:13.335918 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:14.025736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:14.025706 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:14.038650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:14.038603 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:14.803250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:14.803222 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:29:18.048655 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:18.048624 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:22.666352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:22.666316 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:27.543682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:27.543638 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:32.326932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:32.326894 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:33.067380 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:29:33.067345 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:29:33.133277 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:33.133243 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:37.070248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:37.070218 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:40.346901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:40.346861 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:46.755670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:46.755641 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:47.022304 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:47.022230 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:51.229525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:51.229493 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:54.829034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.828998 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd"] Nov 03 21:29:54.834333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.834306 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848518 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848550 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848568 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c141d35b-42f3-42c7-9877-4e918a21b4dc-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848585 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848797 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848752 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrgf\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-kube-api-access-zsrgf\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848802 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848900 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848886 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848937 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.848986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.848961 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.886206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.886184 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d71a74-openshift-default-dockercfg-nb5n6\"" Nov 03 21:29:54.937831 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.937810 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd"] Nov 03 21:29:54.949892 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.949868 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.949994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.949952 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.949994 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950057 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950105 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950282 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c141d35b-42f3-42c7-9877-4e918a21b4dc-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950327 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950369 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrgf\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-kube-api-access-zsrgf\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950375 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950408 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950446 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950498 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.950760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.950733 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.951477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.951453 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c141d35b-42f3-42c7-9877-4e918a21b4dc-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.952442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.952420 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:54.952901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:54.952883 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:55.006369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.006346 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:55.006802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.006785 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrgf\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-kube-api-access-zsrgf\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:55.147107 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.147083 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:55.570181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.570159 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd"] Nov 03 21:29:55.572845 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:29:55.572819 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc141d35b_42f3_42c7_9877_4e918a21b4dc.slice/crio-636ead21949add8cb6d80a4092b86ff5633fbd1c27b7097b1a4823a305ac1f4a WatchSource:0}: Error finding container 636ead21949add8cb6d80a4092b86ff5633fbd1c27b7097b1a4823a305ac1f4a: Status 404 returned error can't find the container with id 636ead21949add8cb6d80a4092b86ff5633fbd1c27b7097b1a4823a305ac1f4a Nov 03 21:29:55.575075 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.575048 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:29:55.575139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.575109 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:29:55.575176 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.575138 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:29:55.927493 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:55.927447 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:56.025229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:56.025197 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" event={"ID":"c141d35b-42f3-42c7-9877-4e918a21b4dc","Type":"ContainerStarted","Data":"1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a"} Nov 03 21:29:56.025439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:56.025249 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" event={"ID":"c141d35b-42f3-42c7-9877-4e918a21b4dc","Type":"ContainerStarted","Data":"636ead21949add8cb6d80a4092b86ff5633fbd1c27b7097b1a4823a305ac1f4a"} Nov 03 21:29:56.145378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:56.144752 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podStartSLOduration=2.144732647 podStartE2EDuration="2.144732647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:56.13929603 +0000 UTC m=+863.750143845" watchObservedRunningTime="2026-11-03 21:29:56.144732647 +0000 UTC m=+863.755580462" Nov 03 21:29:56.148856 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:56.148749 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:29:56.149712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:56.149683 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:29:56.149845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:56.149739 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:29:57.148064 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:57.148028 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:29:57.148528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:57.148091 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:29:58.148052 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:58.148024 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:29:58.148223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:58.148073 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:29:59.148104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:59.148071 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:29:59.148307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:29:59.148144 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:30:00.148146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:00.148108 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:30:00.148336 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:00.148169 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:30:00.653606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:00.653577 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:00.871335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:00.871301 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj"] Nov 03 21:30:00.871526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:00.871503 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" containerID="cri-o://227b71d1ea52a8ffc3ccd8b3a4e421f11c9623976724d26f41ee31be58f8665a" gracePeriod=30 Nov 03 21:30:01.148047 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:01.148011 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:30:01.148227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:01.148067 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 21:30:02.686658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:02.686603 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:30:02.687184 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:02.687161 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:30:02.688059 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:02.688037 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:30:03.067644 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:30:03.067531 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:30:05.535060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:05.535028 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:06.137982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.137908 5931 generic.go:356] "Generic (PLEG): container finished" podID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerID="227b71d1ea52a8ffc3ccd8b3a4e421f11c9623976724d26f41ee31be58f8665a" exitCode=0 Nov 03 21:30:06.137982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.137956 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" event={"ID":"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1","Type":"ContainerDied","Data":"227b71d1ea52a8ffc3ccd8b3a4e421f11c9623976724d26f41ee31be58f8665a"} Nov 03 21:30:06.138171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.137997 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" event={"ID":"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1","Type":"ContainerDied","Data":"29ff9169c186c58817d9d107b410a4e372064c1de6840a238964046ff5e1f566"} Nov 03 21:30:06.138171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.138012 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ff9169c186c58817d9d107b410a4e372064c1de6840a238964046ff5e1f566" Nov 03 21:30:06.139964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.139933 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:30:06.166571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166540 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-socket\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.166702 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166579 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-credential-socket\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.166702 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166622 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-kube-api-access-9m6pw\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-kube-api-access-9m6pw\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.166702 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166667 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-podinfo\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.166880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166701 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-token\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-token\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.166880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166741 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-envoy\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.166880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166798 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-socket" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:06.166880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166858 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-certs\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.167080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166885 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istiod-ca-cert\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.167080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166962 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-data\") pod \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\" (UID: \"38b7fe9e-df1c-4fce-bd82-0a8acaf735b1\") " Nov 03 21:30:06.167189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.166883 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-credential-socket" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:06.167240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.167108 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-certs" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:06.167330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.167310 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-data" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:06.167395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.167376 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.167453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.167402 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.167453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.167417 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.167546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.167510 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istiod-ca-cert" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:06.168924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.168898 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-kube-api-access-9m6pw" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "kube-api-access-9m6pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:06.169418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.169390 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-envoy" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:06.169418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.169401 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-podinfo" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:30:06.169537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.169445 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-token" pod "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" (UID: "38b7fe9e-df1c-4fce-bd82-0a8acaf735b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:06.268160 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.268141 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.268249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.268163 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6pw\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-kube-api-access-9m6pw\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.268249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.268175 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.268249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.268184 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.268249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.268191 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.268249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.268199 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:06.503540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:06.503455 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:07.143689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:07.143659 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj" Nov 03 21:30:07.355350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:07.355321 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj"] Nov 03 21:30:07.357233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:07.357211 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c8qjj"] Nov 03 21:30:08.505507 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:08.505466 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" path="/var/lib/kubelet/pods/38b7fe9e-df1c-4fce-bd82-0a8acaf735b1/volumes" Nov 03 21:30:10.036677 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.036643 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f"] Nov 03 21:30:10.037071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.037059 5931 cpu_manager.go:414] "containerMap: removing container" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" Nov 03 21:30:10.037113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.037072 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" Nov 03 21:30:10.037202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.037190 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b7fe9e-df1c-4fce-bd82-0a8acaf735b1" containerName="istio-proxy" Nov 03 21:30:10.041808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.041789 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.096673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.096646 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f"] Nov 03 21:30:10.103103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.103077 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.103217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.103147 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gmw\" (UniqueName: \"kubernetes.io/projected/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-kube-api-access-f4gmw\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.103336 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.103312 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.126639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.125801 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:30:10.142816 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.142797 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\"" Nov 03 21:30:10.204725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.204698 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.204896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.204745 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gmw\" (UniqueName: \"kubernetes.io/projected/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-kube-api-access-f4gmw\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.204896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.204818 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.205368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.205345 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.207121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.207101 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.225787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.225756 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:10.262354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.262328 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gmw\" (UniqueName: \"kubernetes.io/projected/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-kube-api-access-f4gmw\") pod \"maas-api-revoke-sub-7ae8efbe-rbp6f\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.355409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.355387 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:10.726568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.726524 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f"] Nov 03 21:30:10.728887 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:30:10.728861 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e34ba5_e392_486d_bdbd_b76eb6f28bfb.slice/crio-4cfdca2b18374c33fdda3a671cb9b83d441eb6461cc983a91bd506c9edb5ab3b WatchSource:0}: Error finding container 4cfdca2b18374c33fdda3a671cb9b83d441eb6461cc983a91bd506c9edb5ab3b: Status 404 returned error can't find the container with id 4cfdca2b18374c33fdda3a671cb9b83d441eb6461cc983a91bd506c9edb5ab3b Nov 03 21:30:10.730651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:10.730606 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:30:11.167200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:11.167162 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" event={"ID":"74e34ba5-e392-486d-bdbd-b76eb6f28bfb","Type":"ContainerStarted","Data":"4cfdca2b18374c33fdda3a671cb9b83d441eb6461cc983a91bd506c9edb5ab3b"} Nov 03 21:30:14.977636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:14.976785 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:15.194841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:15.194799 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" event={"ID":"74e34ba5-e392-486d-bdbd-b76eb6f28bfb","Type":"ContainerStarted","Data":"ab8da60bb0734683fe985f408319fb928439a524e92059f9da07234c8bd33f89"} Nov 03 21:30:15.310595 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:15.310545 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" podStartSLOduration=2.5828335620000002 podStartE2EDuration="6.310530343s" totalImagesPullingTime="3.727696781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:15.306773079 +0000 UTC m=+882.917620894" watchObservedRunningTime="2026-11-03 21:30:15.310530343 +0000 UTC m=+882.921378192" Nov 03 21:30:16.202678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:16.202638 5931 generic.go:356] "Generic (PLEG): container finished" podID="74e34ba5-e392-486d-bdbd-b76eb6f28bfb" containerID="ab8da60bb0734683fe985f408319fb928439a524e92059f9da07234c8bd33f89" exitCode=0 Nov 03 21:30:16.203041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:16.202710 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" event={"ID":"74e34ba5-e392-486d-bdbd-b76eb6f28bfb","Type":"ContainerDied","Data":"ab8da60bb0734683fe985f408319fb928439a524e92059f9da07234c8bd33f89"} Nov 03 21:30:16.501586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:16.501506 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:17.334090 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.334066 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:17.490841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.490769 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-openshift-ca\") pod \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " Nov 03 21:30:17.490841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.490814 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-xks-ca\") pod \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " Nov 03 21:30:17.491683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.490846 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-kube-api-access-f4gmw\" (UniqueName: \"kubernetes.io/projected/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-kube-api-access-f4gmw\") pod \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\" (UID: \"74e34ba5-e392-486d-bdbd-b76eb6f28bfb\") " Nov 03 21:30:17.491683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.491159 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-openshift-ca" pod "74e34ba5-e392-486d-bdbd-b76eb6f28bfb" (UID: "74e34ba5-e392-486d-bdbd-b76eb6f28bfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:17.493773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.493746 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-xks-ca" pod "74e34ba5-e392-486d-bdbd-b76eb6f28bfb" (UID: "74e34ba5-e392-486d-bdbd-b76eb6f28bfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:17.493773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.493751 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-kube-api-access-f4gmw" pod "74e34ba5-e392-486d-bdbd-b76eb6f28bfb" (UID: "74e34ba5-e392-486d-bdbd-b76eb6f28bfb"). InnerVolumeSpecName "kube-api-access-f4gmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:17.592572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.592538 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gmw\" (UniqueName: \"kubernetes.io/projected/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-kube-api-access-f4gmw\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:17.592572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.592568 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-openshift-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:17.592572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:17.592578 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e34ba5-e392-486d-bdbd-b76eb6f28bfb-maas-api-xks-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:30:18.215735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:18.215708 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" Nov 03 21:30:18.215887 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:18.215708 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f" event={"ID":"74e34ba5-e392-486d-bdbd-b76eb6f28bfb","Type":"ContainerDied","Data":"4cfdca2b18374c33fdda3a671cb9b83d441eb6461cc983a91bd506c9edb5ab3b"} Nov 03 21:30:18.215887 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:18.215828 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cfdca2b18374c33fdda3a671cb9b83d441eb6461cc983a91bd506c9edb5ab3b" Nov 03 21:30:19.629187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:19.629154 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:24.328922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:24.328890 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:29.156835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:29.156793 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:32.203714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.203683 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5"] Nov 03 21:30:32.204091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.204079 5931 cpu_manager.go:414] "containerMap: removing container" podUID="74e34ba5-e392-486d-bdbd-b76eb6f28bfb" containerName="revoke-keys" Nov 03 21:30:32.204137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.204091 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e34ba5-e392-486d-bdbd-b76eb6f28bfb" containerName="revoke-keys" Nov 03 21:30:32.204205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.204197 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e34ba5-e392-486d-bdbd-b76eb6f28bfb" containerName="revoke-keys" Nov 03 21:30:32.207462 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.207445 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.259406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.259377 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6de8e6-openshift-default-dockercfg-vxss9\"" Nov 03 21:30:32.311398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.311375 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5"] Nov 03 21:30:32.314222 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314203 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ld8c\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-kube-api-access-2ld8c\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314230 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314258 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314333 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572ab12d-96b0-40dd-91c3-67757727d099-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314358 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314414 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314438 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314468 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572ab12d-96b0-40dd-91c3-67757727d099-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.314562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.314538 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.415774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415750 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.415881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415811 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ld8c\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-kube-api-access-2ld8c\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.415881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415829 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.415881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415854 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.415881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415880 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572ab12d-96b0-40dd-91c3-67757727d099-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415901 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415931 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415955 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.415976 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572ab12d-96b0-40dd-91c3-67757727d099-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.416240 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416323 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.416298 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.416355 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.416461 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.416781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.416731 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572ab12d-96b0-40dd-91c3-67757727d099-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.418225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.418205 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.418451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.418434 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572ab12d-96b0-40dd-91c3-67757727d099-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.472470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.472420 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.472551 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.472486 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ld8c\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-kube-api-access-2ld8c\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.570840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.570823 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6de8e6-openshift-default-dockercfg-vxss9\"" Nov 03 21:30:32.580433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.580418 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:32.643865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.643840 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:30:32.644582 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:32.644558 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:30:33.021515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.021487 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5"] Nov 03 21:30:33.023460 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:30:33.023427 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572ab12d_96b0_40dd_91c3_67757727d099.slice/crio-91761efb63783a398040f0dc3d0c014be8e0eb4b3674da497261d9be12e4b03a WatchSource:0}: Error finding container 91761efb63783a398040f0dc3d0c014be8e0eb4b3674da497261d9be12e4b03a: Status 404 returned error can't find the container with id 91761efb63783a398040f0dc3d0c014be8e0eb4b3674da497261d9be12e4b03a Nov 03 21:30:33.025948 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.025910 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:33.026069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.025997 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:33.026069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.026040 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:30:33.072793 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:30:33.072760 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:30:33.298047 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.297951 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" event={"ID":"572ab12d-96b0-40dd-91c3-67757727d099","Type":"ContainerStarted","Data":"891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e"} Nov 03 21:30:33.298047 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.298021 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" event={"ID":"572ab12d-96b0-40dd-91c3-67757727d099","Type":"ContainerStarted","Data":"91761efb63783a398040f0dc3d0c014be8e0eb4b3674da497261d9be12e4b03a"} Nov 03 21:30:33.420638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.418767 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podStartSLOduration=1.418746963 podStartE2EDuration="1.418746963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:33.413037088 +0000 UTC m=+901.023884926" watchObservedRunningTime="2026-11-03 21:30:33.418746963 +0000 UTC m=+901.029594777" Nov 03 21:30:33.581392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.581316 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:33.868647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:33.865225 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:34.582885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:34.582853 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:30:34.583266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:34.582904 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 21:30:35.581419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:35.581389 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:30:35.581596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:35.581439 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 21:30:36.581191 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:36.581158 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:30:36.581604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:36.581205 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 21:30:37.580805 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:37.580772 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:30:37.580980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:37.580830 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 21:30:38.581075 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:38.581044 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:30:38.581461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:38.581096 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 21:30:38.649103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:38.649067 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:39.585269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:39.585242 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:39.585730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:39.585598 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:39.586319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:39.586305 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:30:40.660421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:40.660391 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:40.885469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:40.885439 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:43.265954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:43.265883 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:44.386758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:44.386729 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:47.931785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:47.931753 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:52.851936 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:52.851898 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:57.669907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:30:57.669870 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:01.375213 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.375182 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6"] Nov 03 21:31:01.382292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.382273 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.476763 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.476738 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6"] Nov 03 21:31:01.525920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.525894 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.525931 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.526007 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.526078 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck99j\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-kube-api-access-ck99j\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.526118 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.526143 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.526168 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.526216 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.526287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.526271 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.626722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626666 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.626722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626693 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.626722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626713 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.626911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626734 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.626911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626790 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.626911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626840 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.626911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626868 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.627108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626910 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.627108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.626955 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck99j\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-kube-api-access-ck99j\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.627108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.627060 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.627251 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.627185 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.627489 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.627470 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.627567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.627542 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.627722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.627550 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.628963 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.628941 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.629349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.629333 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.683229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.683207 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.683760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.683740 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck99j\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-kube-api-access-ck99j\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8cfq6\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:01.695142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:01.695122 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:02.120798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.120768 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6"] Nov 03 21:31:02.122744 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:31:02.122717 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00308f0_47ba_4b9b_afb9_edcd6e994a4f.slice/crio-60e8878d51af2f0b39acaa8383a6eac3ad8f2d8656220faabc2470bfe6bb753e WatchSource:0}: Error finding container 60e8878d51af2f0b39acaa8383a6eac3ad8f2d8656220faabc2470bfe6bb753e: Status 404 returned error can't find the container with id 60e8878d51af2f0b39acaa8383a6eac3ad8f2d8656220faabc2470bfe6bb753e Nov 03 21:31:02.125452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.125422 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:02.125559 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.125481 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:02.125559 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.125509 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:02.347565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.347514 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:02.487428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.487347 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" event={"ID":"e00308f0-47ba-4b9b-afb9-edcd6e994a4f","Type":"ContainerStarted","Data":"40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2"} Nov 03 21:31:02.487837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.487441 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" event={"ID":"e00308f0-47ba-4b9b-afb9-edcd6e994a4f","Type":"ContainerStarted","Data":"60e8878d51af2f0b39acaa8383a6eac3ad8f2d8656220faabc2470bfe6bb753e"} Nov 03 21:31:02.604722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.604674 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podStartSLOduration=1.6046600770000001 podStartE2EDuration="1.604660077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:02.602476391 +0000 UTC m=+930.213324206" watchObservedRunningTime="2026-11-03 21:31:02.604660077 +0000 UTC m=+930.215507891" Nov 03 21:31:02.696126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.696102 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:02.995199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:02.995168 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:03.067727 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:31:03.067693 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:31:03.697215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:03.697181 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:31:03.697596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:03.697229 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 21:31:04.696061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:04.696024 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:31:04.696229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:04.696124 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 21:31:05.695911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:05.695870 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:31:05.696325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:05.695940 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 21:31:06.696189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:06.696157 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:31:06.696735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:06.696212 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 21:31:07.052695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:07.052084 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:07.696313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:07.696276 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:31:07.696709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:07.696338 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 21:31:08.696267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:08.696231 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:31:08.696462 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:08.696285 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 21:31:09.695774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:09.695742 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:31:09.695934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:09.695791 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 21:31:10.654202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:10.654166 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:10.845848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:10.845820 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:10.846349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:10.846320 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:10.847215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:10.847195 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:31:10.878171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:10.878147 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:11.735801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:11.735771 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:13.948278 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:13.948243 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:16.453785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.453729 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:16.514436 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.514407 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t"] Nov 03 21:31:16.517952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.517934 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.564011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.563992 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb"] Nov 03 21:31:16.567313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.567287 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586149 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586128 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586164 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586180 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586197 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586338 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xcw7\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-kube-api-access-7xcw7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586411 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586437 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586480 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586543 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586569 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586596 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586672 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586701 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586728 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586798 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.586951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586903 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtck7\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-kube-api-access-qtck7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.586951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586940 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.587067 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.586963 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.666469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.666438 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb"] Nov 03 21:31:16.688267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688242 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.688478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688272 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.688478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688290 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.688478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688311 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.688478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688439 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.688756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688484 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.688756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688512 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.688756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688537 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.688756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688562 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.688756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688604 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.688756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688733 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.688756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688742 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtck7\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-kube-api-access-qtck7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.689124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688800 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.689124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688827 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.689124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688883 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.689124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688930 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.689124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688954 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.689124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.688997 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.689124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.689109 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.689478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.689405 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.689478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.689416 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.690333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.689785 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.690333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.689896 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.690333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.690047 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xcw7\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-kube-api-access-7xcw7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.690333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.690115 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.690333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.690161 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.690685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.690345 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.690685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.690561 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.691120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.691096 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.691474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.691451 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.692505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.692486 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.692571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.692519 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.726572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.726492 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t"] Nov 03 21:31:16.745639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.745570 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.745785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.745760 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtck7\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-kube-api-access-qtck7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.745847 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.745801 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9xjb\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:16.767312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.767292 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xcw7\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-kube-api-access-7xcw7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-7pj5t\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.831182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.831152 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:16.880967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:16.879449 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:17.258825 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.258799 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t"] Nov 03 21:31:17.298041 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:31:17.298015 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1756cafc_d3aa_4c56_a1cc_95006bf629c1.slice/crio-9ea9d2b8a3dfaafc220b1c476c829f594fd429b5c14af41064923256bef023f3 WatchSource:0}: Error finding container 9ea9d2b8a3dfaafc220b1c476c829f594fd429b5c14af41064923256bef023f3: Status 404 returned error can't find the container with id 9ea9d2b8a3dfaafc220b1c476c829f594fd429b5c14af41064923256bef023f3 Nov 03 21:31:17.300239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.300208 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:17.300334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.300270 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:17.300334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.300308 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:17.313982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.313959 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb"] Nov 03 21:31:17.314743 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:31:17.314721 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b66ff23_cdbf_42e6_b567_ca2aba8e82f2.slice/crio-ff3e043ae15be616477f216b1ebc02c663be46acf248fbd37eee701b3d3d314a WatchSource:0}: Error finding container ff3e043ae15be616477f216b1ebc02c663be46acf248fbd37eee701b3d3d314a: Status 404 returned error can't find the container with id ff3e043ae15be616477f216b1ebc02c663be46acf248fbd37eee701b3d3d314a Nov 03 21:31:17.317011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.316982 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:17.317087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.317041 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:17.317087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.317069 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:31:17.582019 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.581934 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" event={"ID":"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2","Type":"ContainerStarted","Data":"f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a"} Nov 03 21:31:17.582466 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.582442 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" event={"ID":"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2","Type":"ContainerStarted","Data":"ff3e043ae15be616477f216b1ebc02c663be46acf248fbd37eee701b3d3d314a"} Nov 03 21:31:17.584213 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.584181 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" event={"ID":"1756cafc-d3aa-4c56-a1cc-95006bf629c1","Type":"ContainerStarted","Data":"5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7"} Nov 03 21:31:17.584331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.584238 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" event={"ID":"1756cafc-d3aa-4c56-a1cc-95006bf629c1","Type":"ContainerStarted","Data":"9ea9d2b8a3dfaafc220b1c476c829f594fd429b5c14af41064923256bef023f3"} Nov 03 21:31:17.701928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.701870 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podStartSLOduration=1.701858003 podStartE2EDuration="1.701858003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:17.697718874 +0000 UTC m=+945.308566720" watchObservedRunningTime="2026-11-03 21:31:17.701858003 +0000 UTC m=+945.312705811" Nov 03 21:31:17.816126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.816078 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podStartSLOduration=1.81606041 podStartE2EDuration="1.81606041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:17.814776029 +0000 UTC m=+945.425623867" watchObservedRunningTime="2026-11-03 21:31:17.81606041 +0000 UTC m=+945.426908225" Nov 03 21:31:17.831474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.831443 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:17.880140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:17.880116 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:18.838634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:18.836600 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:31:18.838634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:18.836699 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:31:18.881498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:18.881456 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:31:18.881663 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:18.881516 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 21:31:19.842638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:19.839812 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:31:19.842638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:19.839876 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:31:19.880252 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:19.880211 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:31:19.880416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:19.880263 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 21:31:20.832707 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:20.832237 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:31:20.832707 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:20.832293 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:31:20.880219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:20.880083 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:31:20.880219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:20.880147 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 21:31:21.618635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:21.614151 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:21.835123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:21.835086 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:31:21.835439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:21.835393 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:31:21.880205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:21.880107 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:31:21.880477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:21.880434 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 21:31:22.837637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:22.833995 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:31:22.837637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:22.834057 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:31:22.891637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:22.891322 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:31:22.891637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:22.891379 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 21:31:23.218637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:23.202805 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:31:23.218637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:23.202885 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:31:23.835411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:23.835015 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:31:23.835411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:23.835073 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:31:23.880020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:23.879967 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:31:23.880260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:23.880235 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 21:31:24.832445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:24.832406 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:31:24.832879 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:24.832482 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 21:31:24.880838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:24.880809 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:31:24.880943 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:24.880865 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 21:31:26.831831 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:26.831790 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:31:26.832243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:26.831857 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:31:26.839144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:26.839114 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:26.881354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:26.881313 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:31:26.881501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:26.881388 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:31:27.873790 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:27.873736 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-7pj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:31:27.874291 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:27.873814 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:31:27.922834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:27.922789 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9xjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:31:27.923007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:27.922846 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:31:28.547423 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:28.547390 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:28.547860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:28.547835 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:28.548595 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:28.548577 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:31:28.602923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:28.602906 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:28.603187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:28.603173 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:28.611213 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:28.611196 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:31:31.665980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:31.665945 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:32.647783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:32.647755 5931 scope.go:122] "RemoveContainer" containerID="227b71d1ea52a8ffc3ccd8b3a4e421f11c9623976724d26f41ee31be58f8665a" Nov 03 21:31:33.068822 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:31:33.068741 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:31:36.330103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:36.330065 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:41.025216 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:41.025188 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:41.248994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:41.248968 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:44.372187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:44.372156 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:45.865206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:45.865179 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:47.358336 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.358292 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj"] Nov 03 21:31:47.362156 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.362140 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.421321 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.421278 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"default-dockercfg-vq2z2\"" Nov 03 21:31:47.466102 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.466072 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"openshift-service-ca.crt\"" Nov 03 21:31:47.470903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.470883 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"kube-root-ca.crt\"" Nov 03 21:31:47.471051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.470978 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"distinct-w1-f463c9-kserve-self-signed-certs\"" Nov 03 21:31:47.473493 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.473467 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj"] Nov 03 21:31:47.498461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.498427 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81838a1c-e57a-49e0-9286-5340e392a979-tls-certs\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.498594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.498509 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-dshm\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.498594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.498564 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc77z\" (UniqueName: \"kubernetes.io/projected/81838a1c-e57a-49e0-9286-5340e392a979-kube-api-access-jc77z\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.498734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.498606 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-home\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.498734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.498681 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-model-cache\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.498840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.498797 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-tmp-dir\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.550458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.550388 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:47.600582 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.600542 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81838a1c-e57a-49e0-9286-5340e392a979-tls-certs\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.600791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.600653 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-dshm\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.600791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.600741 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc77z\" (UniqueName: \"kubernetes.io/projected/81838a1c-e57a-49e0-9286-5340e392a979-kube-api-access-jc77z\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.600925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.600835 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-home\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.600925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.600903 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-model-cache\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.601058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.600957 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-tmp-dir\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.601250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.601226 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-home\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.601250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.601240 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-model-cache\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.601597 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.601579 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-tmp-dir\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.603369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.603347 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81838a1c-e57a-49e0-9286-5340e392a979-tls-certs\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.603477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.603428 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-dshm\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.657428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.657343 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc77z\" (UniqueName: \"kubernetes.io/projected/81838a1c-e57a-49e0-9286-5340e392a979-kube-api-access-jc77z\") pod \"distinct-w1-f463c9-kserve-86f5c7f697-x6ssj\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:47.673806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:47.673781 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:48.048434 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:48.048410 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj"] Nov 03 21:31:48.050724 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:31:48.050693 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81838a1c_e57a_49e0_9286_5340e392a979.slice/crio-8b4ffdb8584f0d2b02a597f650a3e13e27b331595fcebd2c3dba01a1d5449c73 WatchSource:0}: Error finding container 8b4ffdb8584f0d2b02a597f650a3e13e27b331595fcebd2c3dba01a1d5449c73: Status 404 returned error can't find the container with id 8b4ffdb8584f0d2b02a597f650a3e13e27b331595fcebd2c3dba01a1d5449c73 Nov 03 21:31:48.821584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:48.821541 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" event={"ID":"81838a1c-e57a-49e0-9286-5340e392a979","Type":"ContainerStarted","Data":"a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f"} Nov 03 21:31:48.822108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:48.821592 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" event={"ID":"81838a1c-e57a-49e0-9286-5340e392a979","Type":"ContainerStarted","Data":"8b4ffdb8584f0d2b02a597f650a3e13e27b331595fcebd2c3dba01a1d5449c73"} Nov 03 21:31:48.936510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:48.936459 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podStartSLOduration=1.936446119 podStartE2EDuration="1.936446119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:48.9341968 +0000 UTC m=+976.545044615" watchObservedRunningTime="2026-11-03 21:31:48.936446119 +0000 UTC m=+976.547293984" Nov 03 21:31:49.674715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:49.674675 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:50.686533 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:50.686492 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:55.365633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:55.364117 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:57.674244 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:57.674210 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:57.686517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:57.686495 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:31:57.887687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:31:57.887654 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:32:00.035853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:00.035823 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:03.067806 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:32:03.067772 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:32:04.731164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:04.731133 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:09.443261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:09.443227 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:12.509005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.508934 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg"] Nov 03 21:32:12.513515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.513496 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.603833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.603802 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg"] Nov 03 21:32:12.615158 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.615137 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"e2e-conflict-b-33894f13-kserve-self-signed-certs\"" Nov 03 21:32:12.655690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.655666 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqnm\" (UniqueName: \"kubernetes.io/projected/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-kube-api-access-hqqnm\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.655796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.655734 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-home\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.655874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.655857 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tmp-dir\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.655916 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.655884 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tls-certs\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.655987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.655968 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-model-cache\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.656041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.656018 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-dshm\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757424 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-home\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757480 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tmp-dir\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757500 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tls-certs\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757533 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-model-cache\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757557 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-dshm\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757649 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqnm\" (UniqueName: \"kubernetes.io/projected/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-kube-api-access-hqqnm\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757916 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-home\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.757966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757952 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tmp-dir\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.758032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.757984 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-model-cache\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.759904 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.759864 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-dshm\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.760172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.760154 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tls-certs\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.794742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.794718 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:12.813433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.813415 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqnm\" (UniqueName: \"kubernetes.io/projected/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-kube-api-access-hqqnm\") pod \"e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:12.825381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:12.825365 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:13.207334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:13.207298 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg"] Nov 03 21:32:13.211013 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:32:13.210982 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc0ad2b4_9d40_48d1_83f1_0e91f5b0ba61.slice/crio-ddb4edbaa9c91e116b87f9828257f42b8630a33717d4043d1357f02c3d09dad1 WatchSource:0}: Error finding container ddb4edbaa9c91e116b87f9828257f42b8630a33717d4043d1357f02c3d09dad1: Status 404 returned error can't find the container with id ddb4edbaa9c91e116b87f9828257f42b8630a33717d4043d1357f02c3d09dad1 Nov 03 21:32:14.021636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:14.021052 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" event={"ID":"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61","Type":"ContainerStarted","Data":"41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a"} Nov 03 21:32:14.021636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:14.021226 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" event={"ID":"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61","Type":"ContainerStarted","Data":"ddb4edbaa9c91e116b87f9828257f42b8630a33717d4043d1357f02c3d09dad1"} Nov 03 21:32:14.156093 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:14.156036 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" podStartSLOduration=2.156017307 podStartE2EDuration="2.156017307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:14.138913926 +0000 UTC m=+1001.749761742" watchObservedRunningTime="2026-11-03 21:32:14.156017307 +0000 UTC m=+1001.766865122" Nov 03 21:32:14.173196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:14.173144 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:14.413637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:14.411178 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:14.711823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:14.693839 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:14.825740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:14.825700 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:18.021742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.021713 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld"] Nov 03 21:32:18.025405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.025391 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.130932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.130902 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld"] Nov 03 21:32:18.206411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206380 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b6fee0-8552-4713-b61e-35efb29afde6-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206424 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206442 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206535 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206602 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206652 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206671 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kf6m\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-kube-api-access-4kf6m\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206777 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.206925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.206835 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307288 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307323 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307341 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kf6m\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-kube-api-access-4kf6m\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307359 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307394 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307488 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b6fee0-8552-4713-b61e-35efb29afde6-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307534 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307553 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307597 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307687 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307743 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.307923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307890 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.308210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.307984 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.308210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.308139 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b6fee0-8552-4713-b61e-35efb29afde6-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.309835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.309819 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.310023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.310008 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.363490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.363458 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.363586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.363533 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kf6m\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-kube-api-access-4kf6m\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:18.638766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:18.638743 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:19.066248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:19.066221 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld"] Nov 03 21:32:19.067536 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:32:19.067506 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b6fee0_8552_4713_b61e_35efb29afde6.slice/crio-aab53bc0e1034233a149c09881b5e252798e83d7cb4c2e11b2d46cbaa8486c98 WatchSource:0}: Error finding container aab53bc0e1034233a149c09881b5e252798e83d7cb4c2e11b2d46cbaa8486c98: Status 404 returned error can't find the container with id aab53bc0e1034233a149c09881b5e252798e83d7cb4c2e11b2d46cbaa8486c98 Nov 03 21:32:19.069828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:19.069800 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:32:19.069905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:19.069862 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:32:19.069905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:19.069889 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:32:19.132293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:19.132264 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:20.081087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:20.081054 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" event={"ID":"b0b6fee0-8552-4713-b61e-35efb29afde6","Type":"ContainerStarted","Data":"6017118e3fd81f95b606f948793844b09e48db489e02d3a03a8019f9e18151f4"} Nov 03 21:32:20.081461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:20.081106 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" event={"ID":"b0b6fee0-8552-4713-b61e-35efb29afde6","Type":"ContainerStarted","Data":"aab53bc0e1034233a149c09881b5e252798e83d7cb4c2e11b2d46cbaa8486c98"} Nov 03 21:32:20.641749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:20.641702 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:32:20.642055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:20.642020 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:32:20.642466 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:20.642434 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:21.642636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:21.641916 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:32:21.642636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:21.641975 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:32:22.644641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:22.640915 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:32:22.644641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:22.640970 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:32:22.829645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:22.828880 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:22.857542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:22.857512 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:22.971689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:22.971558 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" podStartSLOduration=5.97154386 podStartE2EDuration="5.97154386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:20.251558118 +0000 UTC m=+1007.862405929" watchObservedRunningTime="2026-11-03 21:32:22.97154386 +0000 UTC m=+1010.582391673" Nov 03 21:32:23.114010 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:23.113986 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:32:23.642650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:23.641967 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:32:23.642650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:23.642036 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:32:23.854020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:23.853992 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:24.643501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:24.643469 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:32:24.643804 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:24.643774 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 21:32:26.117279 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:26.115757 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:26.117279 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:26.117012 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:26.125411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:26.125387 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:32:28.862301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:28.862260 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:29.659258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:29.659214 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:33.067568 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:32:33.067532 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:32:33.597008 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:33.596975 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:38.258353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:38.258318 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:41.380008 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:41.379974 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:42.973476 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:42.973448 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs"] Nov 03 21:32:42.977511 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:42.977490 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031450 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031415 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031450 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031453 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b528a122-28ef-4c30-a656-a98ed38cddb6-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031642 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031742 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031711 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031750 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031797 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031818 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrjl\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-kube-api-access-zvrjl\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031853 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.031972 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.031947 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.053472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.053440 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:43.073730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.073690 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs"] Nov 03 21:32:43.132970 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.132943 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.132991 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133099 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133154 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133175 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrjl\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-kube-api-access-zvrjl\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133192 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133260 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133340 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133366 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b528a122-28ef-4c30-a656-a98ed38cddb6-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133543 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133578 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133679 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.133965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133945 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.134014 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.133982 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b528a122-28ef-4c30-a656-a98ed38cddb6-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.135495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.135471 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.135598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.135516 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.191928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.191901 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.192823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.192798 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrjl\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-kube-api-access-zvrjl\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.289808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.289750 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:43.714203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.714180 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs"] Nov 03 21:32:43.716176 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:32:43.716155 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb528a122_28ef_4c30_a656_a98ed38cddb6.slice/crio-5b5beb1452efec6b48f22330608f6dd002ab0caefd47e9c9fd84f504ab0b319d WatchSource:0}: Error finding container 5b5beb1452efec6b48f22330608f6dd002ab0caefd47e9c9fd84f504ab0b319d: Status 404 returned error can't find the container with id 5b5beb1452efec6b48f22330608f6dd002ab0caefd47e9c9fd84f504ab0b319d Nov 03 21:32:43.718242 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.718210 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:32:43.718308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.718274 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:32:43.718308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:43.718301 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:32:44.352779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:44.352737 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" event={"ID":"b528a122-28ef-4c30-a656-a98ed38cddb6","Type":"ContainerStarted","Data":"b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b"} Nov 03 21:32:44.353248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:44.352796 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" event={"ID":"b528a122-28ef-4c30-a656-a98ed38cddb6","Type":"ContainerStarted","Data":"5b5beb1452efec6b48f22330608f6dd002ab0caefd47e9c9fd84f504ab0b319d"} Nov 03 21:32:44.569245 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:44.569196 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" podStartSLOduration=2.5691829630000003 podStartE2EDuration="2.569182963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:44.517792961 +0000 UTC m=+1032.128640775" watchObservedRunningTime="2026-11-03 21:32:44.569182963 +0000 UTC m=+1032.180030777" Nov 03 21:32:45.290650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:45.290601 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:45.292344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:45.292320 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:32:45.292468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:45.292364 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:32:46.290273 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:46.290239 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:32:46.290664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:46.290291 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:32:47.290504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:47.290471 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:32:47.290879 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:47.290522 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:32:47.906936 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:47.906902 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:48.147412 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:48.147381 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:48.290923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:48.290844 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:32:48.290923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:48.290891 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:32:49.290855 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:49.290821 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:32:49.291018 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:49.290870 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 21:32:50.679860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:50.679827 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:50.680470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:50.680448 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:50.681577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:50.681559 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:32:52.740596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:52.740554 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:57.464026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:57.463993 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:57.750968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:32:57.750885 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:01.937393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:01.937362 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:02.185806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:02.185765 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:03.068338 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:33:03.068294 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:33:06.936889 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:06.936851 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:11.632748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:11.632711 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:11.857111 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:11.857071 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:16.360579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:16.360538 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:21.076060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:21.076029 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:21.293512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:21.293481 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:25.930972 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:25.930936 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:26.322178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:26.322109 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg"] Nov 03 21:33:26.419967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:26.419929 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:30.680669 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:30.680635 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:33.067719 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:33:33.067686 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:33:33.604122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.604089 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp"] Nov 03 21:33:33.607933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.607918 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697142 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697196 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697353 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697416 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697463 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697501 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mnz\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-kube-api-access-28mnz\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697521 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b7639-e452-40d5-ab96-20d46bc7a601-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697716 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697702 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.697769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.697728 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799260 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799306 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799386 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799415 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799455 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mnz\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-kube-api-access-28mnz\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799485 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b7639-e452-40d5-ab96-20d46bc7a601-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799574 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799604 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.799812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.799678 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.800130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.800093 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.800332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.800164 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.800332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.800257 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.800517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.800410 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.800872 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.800850 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b7639-e452-40d5-ab96-20d46bc7a601-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.802596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.802434 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.802596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.802491 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.822941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.822911 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp"] Nov 03 21:33:33.857994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.857930 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.858678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.858657 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mnz\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-kube-api-access-28mnz\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:33.921266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:33.921245 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:34.350497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.350466 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp"] Nov 03 21:33:34.351863 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:33:34.351839 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3b7639_e452_40d5_ab96_20d46bc7a601.slice/crio-f3fe7b24d1a5972fe3beda724c52d710f38ab0026376cc6e048b5aa0fecc73d5 WatchSource:0}: Error finding container f3fe7b24d1a5972fe3beda724c52d710f38ab0026376cc6e048b5aa0fecc73d5: Status 404 returned error can't find the container with id f3fe7b24d1a5972fe3beda724c52d710f38ab0026376cc6e048b5aa0fecc73d5 Nov 03 21:33:34.353980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.353948 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:34.354056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.354014 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:34.354056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.354042 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:34.723918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.723844 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" event={"ID":"6c3b7639-e452-40d5-ab96-20d46bc7a601","Type":"ContainerStarted","Data":"83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05"} Nov 03 21:33:34.723918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.723897 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" event={"ID":"6c3b7639-e452-40d5-ab96-20d46bc7a601","Type":"ContainerStarted","Data":"f3fe7b24d1a5972fe3beda724c52d710f38ab0026376cc6e048b5aa0fecc73d5"} Nov 03 21:33:34.889720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.889662 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podStartSLOduration=1.8896454280000001 podStartE2EDuration="1.889645428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:34.887531357 +0000 UTC m=+1082.498379171" watchObservedRunningTime="2026-11-03 21:33:34.889645428 +0000 UTC m=+1082.500493242" Nov 03 21:33:34.922359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:34.922329 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:35.551257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:35.551220 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:35.947870 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:35.947833 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:33:35.948063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:35.947884 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:33:36.147416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:36.147375 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:36.373890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:36.373857 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:36.881875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:36.881836 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:36.944319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:36.944284 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:33:36.944648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:36.944592 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:33:37.922669 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:37.921653 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:33:37.922669 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:37.921720 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:33:38.925644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:38.924916 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:33:38.925644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:38.924969 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:33:39.936642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:39.922917 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:33:39.936642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:39.923006 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:33:40.805162 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:40.805124 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:40.921896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:40.921865 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:33:40.922078 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:40.921919 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 21:33:41.342228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.342119 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" podUID="bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" containerName="main" containerID="cri-o://41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a" gracePeriod=45 Nov 03 21:33:41.608434 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.608368 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:33:41.707040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707001 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-dshm\") pod \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " Nov 03 21:33:41.707235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707053 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-kube-api-access-hqqnm\" (UniqueName: \"kubernetes.io/projected/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-kube-api-access-hqqnm\") pod \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " Nov 03 21:33:41.707235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707103 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-home\") pod \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " Nov 03 21:33:41.707235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707167 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tls-certs\") pod \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " Nov 03 21:33:41.707235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707198 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-model-cache\") pod \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " Nov 03 21:33:41.707442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707375 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-home" pod "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" (UID: "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:41.707521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707503 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tmp-dir\") pod \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\" (UID: \"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61\") " Nov 03 21:33:41.707622 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707584 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-model-cache" pod "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" (UID: "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:41.707815 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.707794 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tmp-dir" pod "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" (UID: "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:41.708203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.708184 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:33:41.708275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.708210 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:33:41.708275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.708224 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:33:41.710738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.710710 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tls-certs" pod "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" (UID: "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:33:41.710851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.710757 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-dshm" pod "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" (UID: "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:41.711101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.711079 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-kube-api-access-hqqnm" pod "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" (UID: "bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61"). InnerVolumeSpecName "kube-api-access-hqqnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:33:41.755865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.755822 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:41.793822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.793793 5931 generic.go:356] "Generic (PLEG): container finished" podID="bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" containerID="41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a" exitCode=0 Nov 03 21:33:41.793973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.793839 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" Nov 03 21:33:41.793973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.793867 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" event={"ID":"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61","Type":"ContainerDied","Data":"41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a"} Nov 03 21:33:41.793973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.793906 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg" event={"ID":"bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61","Type":"ContainerDied","Data":"ddb4edbaa9c91e116b87f9828257f42b8630a33717d4043d1357f02c3d09dad1"} Nov 03 21:33:41.793973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.793928 5931 scope.go:122] "RemoveContainer" containerID="41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a" Nov 03 21:33:41.804112 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.804097 5931 scope.go:122] "RemoveContainer" containerID="41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a" Nov 03 21:33:41.804371 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:33:41.804350 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a\": container with ID starting with 41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a not found: ID does not exist" containerID="41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a" Nov 03 21:33:41.804421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.804379 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a"} err="failed to get container status \"41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a\": rpc error: code = NotFound desc = could not find container \"41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a\": container with ID starting with 41a2ddecbd860696cc2ffeb1da6d649546cea537d520261501a4c6462801c77a not found: ID does not exist" Nov 03 21:33:41.809203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.809187 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:33:41.809262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.809206 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:33:41.809262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.809216 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqnm\" (UniqueName: \"kubernetes.io/projected/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61-kube-api-access-hqqnm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:33:41.926228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.926175 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:41.926435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.926419 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:41.927106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.927089 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:33:41.956527 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.956506 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg"] Nov 03 21:33:41.962058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:41.962030 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-b-33894f13-kserve-776689f55b-7n7tg"] Nov 03 21:33:42.512362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:42.512329 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" path="/var/lib/kubelet/pods/bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61/volumes" Nov 03 21:33:45.649087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:45.649056 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:49.232068 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.232034 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl"] Nov 03 21:33:49.232519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.232507 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" containerName="main" Nov 03 21:33:49.232568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.232521 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" containerName="main" Nov 03 21:33:49.232657 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.232647 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0ad2b4-9d40-48d1-83f1-0e91f5b0ba61" containerName="main" Nov 03 21:33:49.235672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.235656 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.290179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.290159 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-da5f0b-openshift-default-dockercfg-nvv9f\"" Nov 03 21:33:49.293880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.293855 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl"] Nov 03 21:33:49.346710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.346687 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl"] Nov 03 21:33:49.350655 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.350638 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.379909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.379888 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380008 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.379926 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380008 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.379947 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.380074 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380170 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.380158 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.380194 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.380221 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664hj\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-kube-api-access-664hj\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.380298 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.380344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.380321 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.404971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.404950 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f463c9-openshift-default-dockercfg-jbvbq\"" Nov 03 21:33:49.409389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.409369 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl"] Nov 03 21:33:49.480962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.480935 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.480972 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqxq\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-kube-api-access-qfqxq\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481003 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481098 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481121 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481140 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481162 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664hj\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-kube-api-access-664hj\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481221 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481252 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481317 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481365 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481406 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481554 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481579 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481592 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481640 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481658 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481669 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.481798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481728 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481755 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.481980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481805 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.482015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.481994 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.482468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.482420 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.483656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.483635 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.483772 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.483752 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.537341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.537315 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.537417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.537379 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-664hj\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-kube-api-access-664hj\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.548264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.548245 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:49.582749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.582724 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.582845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.582760 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.582845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.582791 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqxq\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-kube-api-access-qfqxq\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.582845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.582822 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.582949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.582865 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.582992 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.582968 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583044 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583014 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583099 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583085 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583113 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583115 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583347 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583464 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583476 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.583835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.583818 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.585184 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.585164 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.585406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.585385 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.640206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.640173 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.640346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.640326 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqxq\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-kube-api-access-qfqxq\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.664198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.664178 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:49.974774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.974749 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl"] Nov 03 21:33:49.975567 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:33:49.975542 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaec1e3_3fc9_49e1_a1ec_1622ef13335f.slice/crio-b1556082b4c1da8861ef98441511a197693a6e43fd5919f70a56be2fc7e2850b WatchSource:0}: Error finding container b1556082b4c1da8861ef98441511a197693a6e43fd5919f70a56be2fc7e2850b: Status 404 returned error can't find the container with id b1556082b4c1da8861ef98441511a197693a6e43fd5919f70a56be2fc7e2850b Nov 03 21:33:49.977723 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.977692 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:49.977799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.977754 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:49.977799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:49.977782 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:50.088491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.088456 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl"] Nov 03 21:33:50.088678 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:33:50.088656 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5244e579_2d83_40a1_9de7_a7f92a5b39a3.slice/crio-c78e272a7da38d2b80e4fd199f140a66123501fbfd2a3bf185ee1aa51acf1c7b WatchSource:0}: Error finding container c78e272a7da38d2b80e4fd199f140a66123501fbfd2a3bf185ee1aa51acf1c7b: Status 404 returned error can't find the container with id c78e272a7da38d2b80e4fd199f140a66123501fbfd2a3bf185ee1aa51acf1c7b Nov 03 21:33:50.091117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.091083 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:50.091221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.091153 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:50.091221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.091196 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:33:50.339387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.339266 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:50.888514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.888449 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" event={"ID":"5244e579-2d83-40a1-9de7-a7f92a5b39a3","Type":"ContainerStarted","Data":"9b483b04d488577e24be34b8484569ab2820fbc649f8bcb914b940a977bd5fa7"} Nov 03 21:33:50.888788 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.888575 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" event={"ID":"5244e579-2d83-40a1-9de7-a7f92a5b39a3","Type":"ContainerStarted","Data":"c78e272a7da38d2b80e4fd199f140a66123501fbfd2a3bf185ee1aa51acf1c7b"} Nov 03 21:33:50.891213 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.891187 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" event={"ID":"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f","Type":"ContainerStarted","Data":"c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618"} Nov 03 21:33:50.891335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:50.891227 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" event={"ID":"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f","Type":"ContainerStarted","Data":"b1556082b4c1da8861ef98441511a197693a6e43fd5919f70a56be2fc7e2850b"} Nov 03 21:33:51.111183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.111131 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podStartSLOduration=2.111104352 podStartE2EDuration="2.111104352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:51.060123535 +0000 UTC m=+1098.670971385" watchObservedRunningTime="2026-11-03 21:33:51.111104352 +0000 UTC m=+1098.721952166" Nov 03 21:33:51.177243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.177122 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podStartSLOduration=2.177108138 podStartE2EDuration="2.177108138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:51.174497666 +0000 UTC m=+1098.785345483" watchObservedRunningTime="2026-11-03 21:33:51.177108138 +0000 UTC m=+1098.787955953" Nov 03 21:33:51.550694 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.550408 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:33:51.554633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.553702 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:51.554633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.553757 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:51.672643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.667955 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:33:51.677648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.676738 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:51.677648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:51.676819 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:52.553032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:52.552988 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:52.553763 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:52.553732 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:52.665059 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:52.665021 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:52.665531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:52.665499 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:53.167638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.163824 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 21:33:53.204644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.202712 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b4xql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:33:53.204644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.202770 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b4xql" podUID="5a40b3c2-8230-4946-a555-132a0de19452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:53.549435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.548991 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:53.549435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.549068 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:53.674407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.673988 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:53.674407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.674047 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:53.707091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.706694 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:53.891239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:53.890497 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:54.168836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:54.168725 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:54.169762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:54.169741 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:33:54.660728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:54.564769 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:54.660728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:54.564849 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:54.707637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:54.666889 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:54.707637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:54.705988 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:54.929221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:54.929127 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:55.085854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.085805 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:55.089964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.089395 5931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:55.177638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.169813 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 21:33:55.342836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.342808 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:33:55.629415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.629306 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:55.629706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.629656 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:55.670795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.670758 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:33:55.678646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.674722 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:55.703759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.674899 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:55.704034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.703996 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:55.710172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.710135 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:55.940700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:55.940549 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:56.104644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.089343 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:56.104644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.089737 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:33:56.178635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.178102 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:56.281712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.280944 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:56.562120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.562035 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:56.562354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.562329 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:56.666986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.666952 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:56.667268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.667243 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:56.685928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.685900 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:33:56.712735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.712698 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 21:33:56.713539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.713519 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:33:56.852256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.850998 5931 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:56.901638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.894720 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8cfq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:33:56.901638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:56.894787 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:57.105693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.093502 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:57.185517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.184769 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:33:57.582039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.581952 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:57.582277 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.582249 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:57.587346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.587257 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 21:33:57.665033 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.664998 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:57.665314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.665283 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:57.677222 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.677199 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:33:57.688643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:57.688431 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:33:58.549389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:58.549356 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:58.549983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:58.549953 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:58.666183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:58.666149 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:58.666436 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:58.666410 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:33:59.549566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:59.549532 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:33:59.549953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:59.549585 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 21:33:59.665185 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:59.665148 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 21:33:59.665367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:33:59.665201 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 21:34:00.186767 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:00.186734 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:00.552512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:00.552435 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:34:00.552980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:00.552748 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:34:00.553576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:00.553556 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:34:01.041535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:01.041504 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:01.608509 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:01.608479 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:34:01.608999 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:01.608983 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:34:01.609680 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:01.609661 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:34:03.069256 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:34:03.069223 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:34:05.743542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:05.743506 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:10.435915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:10.435874 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:13.728642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:13.721658 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s"] Nov 03 21:34:13.732635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:13.731059 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:13.984762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:13.984675 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s"] Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107291 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-kube-api-access-8zrcb\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107368 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b22394b-c087-43f1-8a37-862938a4830b-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107416 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107461 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b22394b-c087-43f1-8a37-862938a4830b-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107503 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107584 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107683 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107721 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.163637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.107748 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213059 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213127 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b22394b-c087-43f1-8a37-862938a4830b-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213169 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213248 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213330 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213368 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213396 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213460 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-kube-api-access-8zrcb\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.214056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.213520 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b22394b-c087-43f1-8a37-862938a4830b-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.217127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.217102 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.252684 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.217510 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.252684 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.217809 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.252684 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.218326 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.252684 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.218332 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b22394b-c087-43f1-8a37-862938a4830b-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.273836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.261127 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b22394b-c087-43f1-8a37-862938a4830b-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.301648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.294448 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.302818 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.302762 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.336851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.336793 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-kube-api-access-8zrcb\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:14.402066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:14.401808 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:15.014158 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.014088 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s"] Nov 03 21:34:15.048105 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:34:15.048075 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b22394b_c087_43f1_8a37_862938a4830b.slice/crio-e55fc69834f5213f20ee4164ad51be9a97e05785b251de652ca8a6cf2af6db13 WatchSource:0}: Error finding container e55fc69834f5213f20ee4164ad51be9a97e05785b251de652ca8a6cf2af6db13: Status 404 returned error can't find the container with id e55fc69834f5213f20ee4164ad51be9a97e05785b251de652ca8a6cf2af6db13 Nov 03 21:34:15.050835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.050801 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:34:15.050934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.050886 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:34:15.050993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.050933 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:34:15.518893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.518854 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:15.740424 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.740393 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" event={"ID":"0b22394b-c087-43f1-8a37-862938a4830b","Type":"ContainerStarted","Data":"8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2"} Nov 03 21:34:15.740642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.740449 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" event={"ID":"0b22394b-c087-43f1-8a37-862938a4830b","Type":"ContainerStarted","Data":"e55fc69834f5213f20ee4164ad51be9a97e05785b251de652ca8a6cf2af6db13"} Nov 03 21:34:15.959375 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:15.959289 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podStartSLOduration=2.95927689 podStartE2EDuration="2.95927689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:15.909623233 +0000 UTC m=+1123.520471039" watchObservedRunningTime="2026-11-03 21:34:15.95927689 +0000 UTC m=+1123.570124704" Nov 03 21:34:16.403731 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:16.403699 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:16.405256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:16.405236 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:34:16.405315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:16.405284 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:34:17.404375 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:17.404340 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:34:17.404802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:17.404398 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:34:18.403810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:18.403775 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:34:18.403973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:18.403827 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:34:19.403776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:19.403743 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:34:19.404157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:19.403792 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:34:19.639448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:19.639248 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:20.403726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:20.403676 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:34:20.403913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:20.403753 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:34:20.477867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:20.477835 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:21.404039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:21.403995 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:34:21.404546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:21.404057 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 21:34:22.407807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:22.407776 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:22.408264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:22.408252 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:22.408955 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:22.408936 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:34:25.143980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:25.143939 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:29.898153 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:29.898114 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:30.311131 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.311055 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv"] Nov 03 21:34:30.315139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.315115 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.409733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.409698 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-home\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.409886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.409749 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tls-certs\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.409958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.409892 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-model-cache\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.410013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.409988 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tmp-dir\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.410094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.410076 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-dshm\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.410132 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.410103 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlvf\" (UniqueName: \"kubernetes.io/projected/858f0ceb-1bfa-4d79-a108-bff88911fdc2-kube-api-access-nwlvf\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.419596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.419576 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"distinct-2-w3-da5f0b-kserve-self-signed-certs\"" Nov 03 21:34:30.422239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.422219 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv"] Nov 03 21:34:30.504324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.504294 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:30.511267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511242 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-model-cache\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511299 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tmp-dir\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511351 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-dshm\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511380 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlvf\" (UniqueName: \"kubernetes.io/projected/858f0ceb-1bfa-4d79-a108-bff88911fdc2-kube-api-access-nwlvf\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511444 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-home\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511486 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tls-certs\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511698 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-model-cache\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511760 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tmp-dir\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.511982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.511952 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-home\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.513788 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.513761 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-dshm\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.513930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.513914 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tls-certs\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.568240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.568182 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlvf\" (UniqueName: \"kubernetes.io/projected/858f0ceb-1bfa-4d79-a108-bff88911fdc2-kube-api-access-nwlvf\") pod \"distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:30.627253 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:30.627235 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:31.000598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:31.000566 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv"] Nov 03 21:34:31.001971 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:34:31.001943 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod858f0ceb_1bfa_4d79_a108_bff88911fdc2.slice/crio-f62ce1c533d0dcd0e414eac354bb19990e2fb9c8233df468631dbefc9461bc6c WatchSource:0}: Error finding container f62ce1c533d0dcd0e414eac354bb19990e2fb9c8233df468631dbefc9461bc6c: Status 404 returned error can't find the container with id f62ce1c533d0dcd0e414eac354bb19990e2fb9c8233df468631dbefc9461bc6c Nov 03 21:34:31.843829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:31.843797 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" event={"ID":"858f0ceb-1bfa-4d79-a108-bff88911fdc2","Type":"ContainerStarted","Data":"3a9453aedbd4dea0a90cd317d552e5f8b79d3fcdd28c8a5de86c1f70f08aec11"} Nov 03 21:34:31.844005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:31.843838 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" event={"ID":"858f0ceb-1bfa-4d79-a108-bff88911fdc2","Type":"ContainerStarted","Data":"f62ce1c533d0dcd0e414eac354bb19990e2fb9c8233df468631dbefc9461bc6c"} Nov 03 21:34:31.965098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:31.965042 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" podStartSLOduration=1.965023135 podStartE2EDuration="1.965023135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:31.961287181 +0000 UTC m=+1139.572134996" watchObservedRunningTime="2026-11-03 21:34:31.965023135 +0000 UTC m=+1139.575870949" Nov 03 21:34:32.709851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:32.709026 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:33.345859 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:34:33.345819 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:34:34.643451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:34.643386 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:39.371373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:39.371337 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:40.628286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:40.628259 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:40.641282 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:40.641259 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:41.004847 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:41.004768 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:34:41.176906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:41.176867 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:42.070148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:42.070117 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:44.031679 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:44.031644 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:48.688433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:48.688392 5931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:34:49.739373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:49.739316 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:54.454547 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:54.454510 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:59.194747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:34:59.194704 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:03.067208 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:35:03.067173 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:35:03.969052 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:03.969021 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:04.409635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.409588 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg"] Nov 03 21:35:04.413504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.413485 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.518396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.518367 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg"] Nov 03 21:35:04.550233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550209 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550252 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550393 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550460 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550439 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550530 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550582 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550558 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550581 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe34529-4554-4a26-b59a-483560b5ba1e-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550676 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4df\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-kube-api-access-5f4df\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.550752 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.550704 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651516 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651497 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651523 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651552 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651566 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651585 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe34529-4554-4a26-b59a-483560b5ba1e-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651644 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4df\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-kube-api-access-5f4df\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651681 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651770 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.651927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651824 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.652094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.651946 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.652094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.652000 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.652188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.652170 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.652382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.652360 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe34529-4554-4a26-b59a-483560b5ba1e-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.652433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.652385 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.654014 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.653991 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.654159 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.654144 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.708057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.707998 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4df\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-kube-api-access-5f4df\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.708347 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.708332 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:04.728450 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:04.728430 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:05.115483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.115460 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg"] Nov 03 21:35:05.116510 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:35:05.116478 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe34529_4554_4a26_b59a_483560b5ba1e.slice/crio-c73d0bcfe0110da4c61d5acc02f92c895d19e254205f176e945052138d210aaf WatchSource:0}: Error finding container c73d0bcfe0110da4c61d5acc02f92c895d19e254205f176e945052138d210aaf: Status 404 returned error can't find the container with id c73d0bcfe0110da4c61d5acc02f92c895d19e254205f176e945052138d210aaf Nov 03 21:35:05.118866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.118836 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:05.118952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.118905 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:05.118952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.118935 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:05.207327 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.207297 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" event={"ID":"dfe34529-4554-4a26-b59a-483560b5ba1e","Type":"ContainerStarted","Data":"16ea3cbd22fcc8435493a8dae87e2fa77cde77593a91480d647870507fe86d02"} Nov 03 21:35:05.207448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.207349 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" event={"ID":"dfe34529-4554-4a26-b59a-483560b5ba1e","Type":"ContainerStarted","Data":"c73d0bcfe0110da4c61d5acc02f92c895d19e254205f176e945052138d210aaf"} Nov 03 21:35:05.325383 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.325333 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podStartSLOduration=1.325318985 podStartE2EDuration="1.325318985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:05.322101291 +0000 UTC m=+1172.932949105" watchObservedRunningTime="2026-11-03 21:35:05.325318985 +0000 UTC m=+1172.936166799" Nov 03 21:35:05.738635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:05.737771 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:06.741636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:06.737087 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:35:06.741636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:06.737165 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:35:07.739635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:07.731895 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:35:07.739635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:07.731961 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:35:08.736641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:08.728883 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:35:08.736641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:08.728947 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:35:08.834049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:08.834014 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:09.746752 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:09.746716 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:35:09.747171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:09.746777 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:35:10.738502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:10.738468 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:35:10.738793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:10.738764 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:35:11.730149 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:11.730050 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:35:11.730766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:11.730718 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 21:35:13.727926 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:13.727892 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:13.777751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:13.777719 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:35:13.777905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:13.777786 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:13.883207 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:13.883182 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:13.883682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:13.883647 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:13.884522 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:13.884502 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:35:17.142068 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:17.142033 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:17.789736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:17.789703 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:18.045183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:18.045119 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f"] Nov 03 21:35:18.094494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:18.094463 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae8efbe-rbp6f"] Nov 03 21:35:18.254132 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:18.254100 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:18.506125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:18.506099 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e34ba5-e392-486d-bdbd-b76eb6f28bfb" path="/var/lib/kubelet/pods/74e34ba5-e392-486d-bdbd-b76eb6f28bfb/volumes" Nov 03 21:35:18.652185 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:18.652142 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:19.494825 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.494794 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792"] Nov 03 21:35:19.499020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.498986 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.558903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.558877 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792"] Nov 03 21:35:19.571593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.571568 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bffbc6-20e7-4e8b-b76f-741a027436c9-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.571736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.571600 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.571736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.571637 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.571850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.571813 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.572023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.572006 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.572091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.572030 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slj5m\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-kube-api-access-slj5m\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.572145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.572127 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.572202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.572147 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.572262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.572223 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673348 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673376 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673398 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673441 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bffbc6-20e7-4e8b-b76f-741a027436c9-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673456 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673476 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673536 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673667 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673701 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slj5m\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-kube-api-access-slj5m\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.673836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673795 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.674035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.673907 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.674035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.674023 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.674129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.674034 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.674241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.674223 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bffbc6-20e7-4e8b-b76f-741a027436c9-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.676049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.676029 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.676222 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.676200 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.729957 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.729935 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slj5m\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-kube-api-access-slj5m\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.730046 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.729985 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:19.812777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:19.812722 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:20.241245 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:20.241212 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792"] Nov 03 21:35:20.281353 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:35:20.281328 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bffbc6_20e7_4e8b_b76f_741a027436c9.slice/crio-511243cb50e4ac2de30a81d3d5b5172e011ed6687f18b8a767150a47cb10b5be WatchSource:0}: Error finding container 511243cb50e4ac2de30a81d3d5b5172e011ed6687f18b8a767150a47cb10b5be: Status 404 returned error can't find the container with id 511243cb50e4ac2de30a81d3d5b5172e011ed6687f18b8a767150a47cb10b5be Nov 03 21:35:20.283598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:20.283563 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:20.283721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:20.283651 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:20.283721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:20.283681 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:20.352908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:20.352882 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" event={"ID":"09bffbc6-20e7-4e8b-b76f-741a027436c9","Type":"ContainerStarted","Data":"511243cb50e4ac2de30a81d3d5b5172e011ed6687f18b8a767150a47cb10b5be"} Nov 03 21:35:21.361164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:21.361130 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" event={"ID":"09bffbc6-20e7-4e8b-b76f-741a027436c9","Type":"ContainerStarted","Data":"da8587b75d8e48c50d89966c2fdc7257921b3ff5fccb3a68e008c86509a5bd92"} Nov 03 21:35:21.534636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:21.527397 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podStartSLOduration=2.527378242 podStartE2EDuration="2.527378242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:21.526910164 +0000 UTC m=+1189.137757990" watchObservedRunningTime="2026-11-03 21:35:21.527378242 +0000 UTC m=+1189.138226057" Nov 03 21:35:21.813052 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:21.812977 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:21.814498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:21.814470 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:35:21.814645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:21.814523 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 21:35:22.239638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:22.234973 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:22.555882 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:22.555810 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:22.813997 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:22.813921 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:35:22.813997 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:22.813970 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 21:35:23.779021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:23.778940 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:23.813734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:23.813707 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:35:23.813904 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:23.813755 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 21:35:24.813213 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:24.813179 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:35:24.813584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:24.813229 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 21:35:25.813150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:25.813111 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:35:25.813340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:25.813177 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 21:35:26.673258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:26.673226 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:26.813890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:26.813860 5931 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:35:26.814359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:26.813923 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 21:35:26.919601 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:26.919570 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:27.328946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:27.328909 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:27.569028 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:27.568999 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:27.817349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:27.817326 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:27.817720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:27.817581 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:27.818379 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:27.818363 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:35:28.657329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:28.657286 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:28.883352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:28.883305 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:29.109219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.109167 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:29.805912 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.805880 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb"] Nov 03 21:35:29.809922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.809904 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:29.910120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.910087 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-dshm\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:29.910531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.910139 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30da697-3933-4894-8f70-da325802d8b6-tls-certs\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:29.910531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.910370 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-tmp-dir\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:29.910531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.910422 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-model-cache\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:29.910531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.910517 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-home\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:29.910727 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.910546 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtql\" (UniqueName: \"kubernetes.io/projected/c30da697-3933-4894-8f70-da325802d8b6-kube-api-access-cwtql\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:29.911792 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.911775 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"kube-root-ca.crt\"" Nov 03 21:35:29.911914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.911819 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"openshift-service-ca.crt\"" Nov 03 21:35:29.911914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.911867 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"default-dockercfg-xz5mb\"" Nov 03 21:35:29.912008 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.911942 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"distinct-2-w2-d71a74-kserve-self-signed-certs\"" Nov 03 21:35:29.912964 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:29.912947 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb"] Nov 03 21:35:30.012154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.012118 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-tmp-dir\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.012366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.012183 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-model-cache\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.012366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.012229 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-home\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.012366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.012259 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtql\" (UniqueName: \"kubernetes.io/projected/c30da697-3933-4894-8f70-da325802d8b6-kube-api-access-cwtql\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.012528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.012366 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-dshm\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.012528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.012406 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30da697-3933-4894-8f70-da325802d8b6-tls-certs\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.012838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.012809 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-tmp-dir\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.013087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.013070 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-home\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.013319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.013295 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-model-cache\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.015755 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.015732 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-dshm\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.015879 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.015741 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30da697-3933-4894-8f70-da325802d8b6-tls-certs\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.040468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.040393 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:30.078705 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.078653 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtql\" (UniqueName: \"kubernetes.io/projected/c30da697-3933-4894-8f70-da325802d8b6-kube-api-access-cwtql\") pod \"distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.122448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.122423 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:30.511766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:30.511742 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb"] Nov 03 21:35:30.513807 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:35:30.513775 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30da697_3933_4894_8f70_da325802d8b6.slice/crio-6095fcd70c11c0656062cc0437c7dbd6f81342a56bb3544e76a21c355539d499 WatchSource:0}: Error finding container 6095fcd70c11c0656062cc0437c7dbd6f81342a56bb3544e76a21c355539d499: Status 404 returned error can't find the container with id 6095fcd70c11c0656062cc0437c7dbd6f81342a56bb3544e76a21c355539d499 Nov 03 21:35:31.591448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:31.591407 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" event={"ID":"c30da697-3933-4894-8f70-da325802d8b6","Type":"ContainerStarted","Data":"145d6829826c48cc42b13ee18cefc850588e63cfca9da447d3b697070b4e1c06"} Nov 03 21:35:31.612829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:31.612797 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" event={"ID":"c30da697-3933-4894-8f70-da325802d8b6","Type":"ContainerStarted","Data":"6095fcd70c11c0656062cc0437c7dbd6f81342a56bb3544e76a21c355539d499"} Nov 03 21:35:31.740110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:31.740028 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" podStartSLOduration=2.740009421 podStartE2EDuration="2.740009421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:31.73701717 +0000 UTC m=+1199.347864987" watchObservedRunningTime="2026-11-03 21:35:31.740009421 +0000 UTC m=+1199.350857238" Nov 03 21:35:31.966694 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:31.965654 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:32.008534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:32.003105 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:32.128641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:32.125740 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:32.180660 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:32.179687 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:32.591634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:32.591497 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:32.689838 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:32.688411 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 21:35:33.008801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:33.008699 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:33.009379 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:33.009353 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 21:35:33.035286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:33.035258 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:35:33.112914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:33.112885 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:35:33.590571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:33.590526 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 21:35:33.705669 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:33.695949 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:33.734750 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:33.734722 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:35:33.739924 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:35:33.739890 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:35:34.148853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:34.143486 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 21:35:34.220642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:34.210649 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:34.673647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:34.671699 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 21:35:34.673647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:34.671985 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 21:35:34.738638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:34.735751 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 21:35:34.975757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:34.975625 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:35.129278 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.129226 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr"] Nov 03 21:35:35.189795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.159866 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 21:35:35.189795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.173792 5931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 21:35:35.210632 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.210146 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.210895 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.210862 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" podUID="d2e0fd94-4550-4065-b573-db223a11f0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:35:35.227343 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.223231 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.254710 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.254774 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.254833 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.254886 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqv6s\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-kube-api-access-vqv6s\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.254921 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.254962 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.255060 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.255117 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.256232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.255200 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.297340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.297313 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr"] Nov 03 21:35:35.322577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.322549 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l" Nov 03 21:35:35.366199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.366164 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.366508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.366485 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.366694 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.366677 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.366860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.366841 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.367030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.367008 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqv6s\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-kube-api-access-vqv6s\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.367163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.367146 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.367297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.367281 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.367425 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.367396 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.367605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.367588 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.367816 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.367798 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.368393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.368364 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.368860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.368837 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.370142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.369247 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.370258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.367707 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.390036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.390015 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:35:35.409588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.409565 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:35:35.411913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.411861 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.413140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.412441 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 21:35:35.418424 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.418396 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.430076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.430052 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.430436 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.430052 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqv6s\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-kube-api-access-vqv6s\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:35.604977 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:35.604944 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:36.051043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.050961 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr"] Nov 03 21:35:36.092657 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:35:36.092629 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaec8e7a_caa1_4f31_953c_800b49b5f7c0.slice/crio-0a6ddd458fd1bc28b849066fc5abfb7430392241a3dc9f4f574043a6e2d823b9 WatchSource:0}: Error finding container 0a6ddd458fd1bc28b849066fc5abfb7430392241a3dc9f4f574043a6e2d823b9: Status 404 returned error can't find the container with id 0a6ddd458fd1bc28b849066fc5abfb7430392241a3dc9f4f574043a6e2d823b9 Nov 03 21:35:36.095043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.095009 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:36.095150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.095090 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:36.095150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.095136 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:35:36.314566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.314478 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" event={"ID":"eaec8e7a-caa1-4f31-953c-800b49b5f7c0","Type":"ContainerStarted","Data":"ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321"} Nov 03 21:35:36.314566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.314543 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" event={"ID":"eaec8e7a-caa1-4f31-953c-800b49b5f7c0","Type":"ContainerStarted","Data":"0a6ddd458fd1bc28b849066fc5abfb7430392241a3dc9f4f574043a6e2d823b9"} Nov 03 21:35:36.431520 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.431466 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podStartSLOduration=2.431444061 podStartE2EDuration="2.431444061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:36.428330882 +0000 UTC m=+1204.039178698" watchObservedRunningTime="2026-11-03 21:35:36.431444061 +0000 UTC m=+1204.042291920" Nov 03 21:35:36.605320 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:36.605290 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:37.606626 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:37.606578 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 21:35:37.607074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:37.606647 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 21:35:38.606140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:38.606105 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 21:35:38.606310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:38.606156 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 21:35:39.605842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:39.605809 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 21:35:39.606306 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:39.605863 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 21:35:39.806659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:39.806596 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:40.144367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:40.143663 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:40.164166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:40.164131 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:40.377503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:40.377477 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 21:35:40.605962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:40.605928 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 21:35:40.606533 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:40.606489 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 21:35:41.605842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:41.605811 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 21:35:41.606004 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:41.605865 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 21:35:42.426930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:42.426892 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:42.605378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:42.605345 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 21:35:42.605531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:42.605389 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 21:35:43.609125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:43.609100 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:43.609573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:43.609386 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:43.610098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:43.610083 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:35:44.465039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:44.464992 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:46.653990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:46.653956 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:49.616280 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:49.616242 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:52.871201 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:52.871155 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:53.996389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:53.996354 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9"] Nov 03 21:35:54.000028 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.000012 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.051804 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.051781 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c1662df5-serving-cert\"" Nov 03 21:35:54.104265 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.104240 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9"] Nov 03 21:35:54.129892 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.129856 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.130031 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.130013 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-metrics-certs\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.130089 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.130044 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-openshift-service-ca\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.130261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.130241 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrbz\" (UniqueName: \"kubernetes.io/projected/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-kube-api-access-vhrbz\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.231246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.231215 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-metrics-certs\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.231468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.231253 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-openshift-service-ca\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.231468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.231342 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrbz\" (UniqueName: \"kubernetes.io/projected/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-kube-api-access-vhrbz\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.231468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.231360 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.231468 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:35:54.231464 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-serving-cert: secret "maas-api-e2e-mt-c1662df5-serving-cert" not found Nov 03 21:35:54.231734 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:35:54.231515 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls podName:b0fda6a1-9f90-4049-99c8-e102e2aecbb8 nodeName:}" failed. No retries permitted until 2026-11-03 21:35:54.731501107 +0000 UTC m=+1222.342348914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls") pod "maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" (UID: "b0fda6a1-9f90-4049-99c8-e102e2aecbb8") : secret "maas-api-e2e-mt-c1662df5-serving-cert" not found Nov 03 21:35:54.231968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.231949 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-openshift-service-ca\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.233725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.233704 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-metrics-certs\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.293242 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.293165 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrbz\" (UniqueName: \"kubernetes.io/projected/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-kube-api-access-vhrbz\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.402761 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.402725 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:54.641363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.641335 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:54.737993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.737971 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.740390 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.740365 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:54.894435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.894383 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:54.915595 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:54.915579 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:55.291654 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:55.291629 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9"] Nov 03 21:35:55.293790 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:35:55.293762 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fda6a1_9f90_4049_99c8_e102e2aecbb8.slice/crio-d49957b3451c6cf3fdd0e21a04bb4eade8405fab0df6bb619cf837f889106cd8 WatchSource:0}: Error finding container d49957b3451c6cf3fdd0e21a04bb4eade8405fab0df6bb619cf837f889106cd8: Status 404 returned error can't find the container with id d49957b3451c6cf3fdd0e21a04bb4eade8405fab0df6bb619cf837f889106cd8 Nov 03 21:35:55.295211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:55.295195 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:35:55.657271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:55.657240 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" event={"ID":"b0fda6a1-9f90-4049-99c8-e102e2aecbb8","Type":"ContainerStarted","Data":"d49957b3451c6cf3fdd0e21a04bb4eade8405fab0df6bb619cf837f889106cd8"} Nov 03 21:35:56.664762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:56.664728 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" event={"ID":"b0fda6a1-9f90-4049-99c8-e102e2aecbb8","Type":"ContainerStarted","Data":"ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed"} Nov 03 21:35:56.665138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:56.664900 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:35:56.780468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:56.780425 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" podStartSLOduration=2.883874671 podStartE2EDuration="3.780409394s" totalImagesPullingTime="896.534723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:56.779159086 +0000 UTC m=+1224.390006900" watchObservedRunningTime="2026-11-03 21:35:56.780409394 +0000 UTC m=+1224.391257210" Nov 03 21:35:58.900922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:58.900890 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:59.148065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:59.148030 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:59.377024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:35:59.376991 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:02.676395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:02.676364 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:36:03.069541 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:36:03.069450 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:36:04.043031 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:04.042991 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:08.743180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:08.743141 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:08.970452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:08.970419 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:09.828045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:09.827948 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:10.202705 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:10.181089 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:13.732709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:13.732663 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:18.449241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:18.449202 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:23.232923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:23.232880 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:27.898492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:27.898457 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:31.572071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:31.572033 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:32.949639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:32.947046 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:33.164936 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:36:33.164896 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:36:33.235137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:33.235061 5931 scope.go:122] "RemoveContainer" containerID="ab8da60bb0734683fe985f408319fb928439a524e92059f9da07234c8bd33f89" Nov 03 21:36:37.453633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:37.453573 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:37.680565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:37.680534 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:38.043814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:38.043781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:38.535286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:38.535249 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:42.745967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:42.745849 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:42.984114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:42.984082 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:43.379085 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.379050 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w"] Nov 03 21:36:43.384430 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.384407 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.405820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.405789 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9234ec9-004d-4ffe-8001-9118274d8894-tls-certs\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.405965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.405842 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-dshm\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.405965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.405895 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8r8d\" (UniqueName: \"kubernetes.io/projected/a9234ec9-004d-4ffe-8001-9118274d8894-kube-api-access-g8r8d\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.406098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.406077 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-home\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.406211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.406109 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-model-cache\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.406211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.406157 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-tmp-dir\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.438667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.438630 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w"] Nov 03 21:36:43.484703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.484680 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c1662df5\"/\"default-dockercfg-smjjr\"" Nov 03 21:36:43.484703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.484688 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c1662df5\"/\"kube-root-ca.crt\"" Nov 03 21:36:43.486105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.486086 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c1662df5\"/\"openshift-service-ca.crt\"" Nov 03 21:36:43.486105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.486095 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c1662df5\"/\"e2e-lifecycle-model-c1662df5-kserve-self-signed-certs\"" Nov 03 21:36:43.507280 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.507258 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9234ec9-004d-4ffe-8001-9118274d8894-tls-certs\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.507396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.507290 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-dshm\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.507396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.507327 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8r8d\" (UniqueName: \"kubernetes.io/projected/a9234ec9-004d-4ffe-8001-9118274d8894-kube-api-access-g8r8d\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.507507 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.507428 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-home\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.510571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.507453 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-model-cache\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.510571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.507947 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-tmp-dir\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.510571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.508304 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-tmp-dir\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.514631 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.511266 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9234ec9-004d-4ffe-8001-9118274d8894-tls-certs\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.518466 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.518440 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-model-cache\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.518664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.518645 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-home\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.519393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.519225 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-dshm\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.583303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.583264 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:43.590747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.590715 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8r8d\" (UniqueName: \"kubernetes.io/projected/a9234ec9-004d-4ffe-8001-9118274d8894-kube-api-access-g8r8d\") pod \"e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:43.696263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:43.696181 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:44.074106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:44.074078 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w"] Nov 03 21:36:44.075492 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:36:44.075462 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9234ec9_004d_4ffe_8001_9118274d8894.slice/crio-da2517f08bfd6ab82206508116254fc724e99aeca1900b827168a88a2dd30198 WatchSource:0}: Error finding container da2517f08bfd6ab82206508116254fc724e99aeca1900b827168a88a2dd30198: Status 404 returned error can't find the container with id da2517f08bfd6ab82206508116254fc724e99aeca1900b827168a88a2dd30198 Nov 03 21:36:45.052178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:45.052148 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" event={"ID":"a9234ec9-004d-4ffe-8001-9118274d8894","Type":"ContainerStarted","Data":"2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e"} Nov 03 21:36:45.052335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:45.052189 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" event={"ID":"a9234ec9-004d-4ffe-8001-9118274d8894","Type":"ContainerStarted","Data":"da2517f08bfd6ab82206508116254fc724e99aeca1900b827168a88a2dd30198"} Nov 03 21:36:45.167757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:45.167702 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" podStartSLOduration=2.167689782 podStartE2EDuration="2.167689782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:45.16748095 +0000 UTC m=+1272.778328748" watchObservedRunningTime="2026-11-03 21:36:45.167689782 +0000 UTC m=+1272.778537592" Nov 03 21:36:45.696296 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:45.696267 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:47.558277 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:47.558209 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:52.448833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:52.448794 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:53.696698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:53.696669 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:53.708573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:53.708552 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:54.157875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:54.157847 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:36:57.248770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:57.248727 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:59.403758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:59.403724 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:59.632144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:36:59.632108 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:01.930510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:01.930472 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:03.068430 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:37:03.068393 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:37:03.137792 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:03.137770 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:03.738921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:03.738867 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:03.960120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:03.960084 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:04.474665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:04.474603 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:05.680693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.680662 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt"] Nov 03 21:37:05.685350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.685324 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.739783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.739762 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c5025c-openshift-default-dockercfg-zt76q\"" Nov 03 21:37:05.793155 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.793125 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt"] Nov 03 21:37:05.821661 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.821582 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-credential-socket\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.821920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.821814 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-data\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.821920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.821852 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-certs\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.821920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.821905 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lqxp\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-kube-api-access-2lqxp\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.822197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.821936 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-envoy\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.822197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.821973 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-socket\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.822197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.822058 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istiod-ca-cert\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.822197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.822134 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-podinfo\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.822197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.822160 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-token\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.862464 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.862425 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:05.923948 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.923916 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-data\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.923957 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-certs\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924020 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lqxp\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-kube-api-access-2lqxp\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924047 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-envoy\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924085 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-socket\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924221 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istiod-ca-cert\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924308 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-podinfo\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924334 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-token\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924342 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-data\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924427 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-credential-socket\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924741 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-socket\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924753 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-certs\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.924974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.924957 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-credential-socket\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.925367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.925344 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istiod-ca-cert\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.927199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.927165 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-envoy\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.927395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.927375 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-podinfo\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.980316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.980261 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-token\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:05.980491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:05.980472 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lqxp\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-kube-api-access-2lqxp\") pod \"e2e-api-a-c5025c-openshift-default-8964458d9-w25qt\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:06.003781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:06.003753 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:06.110049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:06.109918 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:06.457712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:06.457686 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt"] Nov 03 21:37:06.459957 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:37:06.459924 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4287f3_df53_47db_874c_3e8d3bfd92f6.slice/crio-59a020874888d0e0d051eae778958a6647560c79b77bd9a4bdebfa2669759bc5 WatchSource:0}: Error finding container 59a020874888d0e0d051eae778958a6647560c79b77bd9a4bdebfa2669759bc5: Status 404 returned error can't find the container with id 59a020874888d0e0d051eae778958a6647560c79b77bd9a4bdebfa2669759bc5 Nov 03 21:37:06.462272 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:06.462229 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:06.462325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:06.462308 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:06.462363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:06.462353 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:06.807459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:06.807324 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:07.247188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:07.247156 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" event={"ID":"6d4287f3-df53-47db-874c-3e8d3bfd92f6","Type":"ContainerStarted","Data":"8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271"} Nov 03 21:37:07.247341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:07.247208 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" event={"ID":"6d4287f3-df53-47db-874c-3e8d3bfd92f6","Type":"ContainerStarted","Data":"59a020874888d0e0d051eae778958a6647560c79b77bd9a4bdebfa2669759bc5"} Nov 03 21:37:07.362478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:07.362417 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podStartSLOduration=2.3624037270000002 podStartE2EDuration="2.362403727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:07.360918292 +0000 UTC m=+1294.971766106" watchObservedRunningTime="2026-11-03 21:37:07.362403727 +0000 UTC m=+1294.973251541" Nov 03 21:37:08.004051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:08.004021 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:08.009874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:08.009852 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:08.179825 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:08.179788 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:08.255438 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:08.255375 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:08.256221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:08.256198 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:37:08.425373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:08.425316 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:11.398790 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:11.398760 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:11.629916 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:11.629881 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:11.904642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:11.903521 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:12.185355 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:12.185279 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:13.163777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.163750 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5"] Nov 03 21:37:13.168323 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.168301 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.213123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.213104 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv"] Nov 03 21:37:13.216806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.216791 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.312040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.312000 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv"] Nov 03 21:37:13.315350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315321 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315375 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315556 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315511 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315556 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315548 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315573 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315668 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315810 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzglm\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-kube-api-access-nzglm\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315873 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de70907b-7e1b-462e-beb9-50b66b931bf9-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.315967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.315949 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.405352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.405321 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5"] Nov 03 21:37:13.417314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417246 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.417314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417309 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzglm\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-kube-api-access-nzglm\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.417506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417394 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de70907b-7e1b-462e-beb9-50b66b931bf9-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.417506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417427 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.417506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417472 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.417696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417548 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkvx\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-kube-api-access-mwkvx\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.417696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417587 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.417696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417630 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.417857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417708 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.417857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417753 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.417857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417807 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417915 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.418013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.417963 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418024 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418049 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.418163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418089 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418113 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418146 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c895147-e359-4fce-ab8a-f58a70f58b82-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.418443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418229 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418427 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de70907b-7e1b-462e-beb9-50b66b931bf9-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418628 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418765 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.418895 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.418872 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.421135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.421100 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.421548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.421528 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.475267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.475234 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.475379 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.475344 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzglm\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-kube-api-access-nzglm\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.482122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.482102 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:13.519353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519325 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519395 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519439 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkvx\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-kube-api-access-mwkvx\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519467 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519493 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519553 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519671 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519759 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519786 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519818 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519858 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c895147-e359-4fce-ab8a-f58a70f58b82-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.519991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.519952 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.520229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.520207 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.520545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.520527 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c895147-e359-4fce-ab8a-f58a70f58b82-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.522189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.522170 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.523733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.523717 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.581817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.581786 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkvx\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-kube-api-access-mwkvx\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.581998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.581974 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.828648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.828556 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:13.925578 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:37:13.925546 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde70907b_7e1b_462e_beb9_50b66b931bf9.slice/crio-e90131fc6df54ebf4e87551b4ee7d87a6a7a3e1131aac0bce37730f30ba681bf WatchSource:0}: Error finding container e90131fc6df54ebf4e87551b4ee7d87a6a7a3e1131aac0bce37730f30ba681bf: Status 404 returned error can't find the container with id e90131fc6df54ebf4e87551b4ee7d87a6a7a3e1131aac0bce37730f30ba681bf Nov 03 21:37:13.926395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.926364 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5"] Nov 03 21:37:13.928217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.928180 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:13.928300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.928261 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:13.928338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:13.928301 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:14.206221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.206196 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv"] Nov 03 21:37:14.208176 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:37:14.208149 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c895147_e359_4fce_ab8a_f58a70f58b82.slice/crio-3c829ce08f4ab85ffa51d2397845f995c285c0b8b77d89bd6f99775f223f42d2 WatchSource:0}: Error finding container 3c829ce08f4ab85ffa51d2397845f995c285c0b8b77d89bd6f99775f223f42d2: Status 404 returned error can't find the container with id 3c829ce08f4ab85ffa51d2397845f995c285c0b8b77d89bd6f99775f223f42d2 Nov 03 21:37:14.211003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.210967 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:14.211104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.211046 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:14.211104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.211089 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:37:14.321664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.321629 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" event={"ID":"0c895147-e359-4fce-ab8a-f58a70f58b82","Type":"ContainerStarted","Data":"f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb"} Nov 03 21:37:14.321841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.321690 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" event={"ID":"0c895147-e359-4fce-ab8a-f58a70f58b82","Type":"ContainerStarted","Data":"3c829ce08f4ab85ffa51d2397845f995c285c0b8b77d89bd6f99775f223f42d2"} Nov 03 21:37:14.324140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.324111 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" event={"ID":"de70907b-7e1b-462e-beb9-50b66b931bf9","Type":"ContainerStarted","Data":"5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567"} Nov 03 21:37:14.324260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.324176 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" event={"ID":"de70907b-7e1b-462e-beb9-50b66b931bf9","Type":"ContainerStarted","Data":"e90131fc6df54ebf4e87551b4ee7d87a6a7a3e1131aac0bce37730f30ba681bf"} Nov 03 21:37:14.464420 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.464322 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podStartSLOduration=1.46430718 podStartE2EDuration="1.46430718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:14.462770381 +0000 UTC m=+1302.073618197" watchObservedRunningTime="2026-11-03 21:37:14.46430718 +0000 UTC m=+1302.075154993" Nov 03 21:37:14.482847 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.482817 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:14.484267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.484243 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 21:37:14.484404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.484292 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 21:37:14.581502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.581443 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podStartSLOduration=1.581428162 podStartE2EDuration="1.581428162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:14.57884998 +0000 UTC m=+1302.189697796" watchObservedRunningTime="2026-11-03 21:37:14.581428162 +0000 UTC m=+1302.192276022" Nov 03 21:37:14.832710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:14.830692 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:15.483170 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:15.483135 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 21:37:15.483648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:15.483187 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 21:37:15.836461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:15.836371 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:37:15.836751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:15.836722 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:37:16.483389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:16.483356 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 21:37:16.483983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:16.483951 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 21:37:16.672644 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:16.662365 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:16.834695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:16.832999 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:37:16.834695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:16.833061 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:37:16.900535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:16.900195 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:17.136577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:17.136500 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:17.483571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:17.483487 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 21:37:17.483571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:17.483543 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 21:37:17.832692 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:17.831947 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:37:17.832692 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:17.832009 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:37:18.482982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:18.482945 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 21:37:18.484986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:18.482997 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 21:37:18.842048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:18.842021 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:37:18.842307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:18.842283 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:37:19.482862 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:19.482821 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 21:37:19.483044 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:19.482898 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 21:37:19.847106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:19.847065 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:37:19.847665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:19.847134 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:37:20.482713 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:20.482671 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 21:37:20.482906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:20.482733 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 21:37:20.829787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:20.829704 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:37:20.829787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:20.829762 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 21:37:21.488504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:21.488475 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:21.489014 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:21.488884 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:21.489685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:21.489668 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:37:21.835348 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:21.835269 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:21.835678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:21.835659 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:21.836319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:21.836299 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:37:22.026802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:22.026774 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:26.643588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:26.643549 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:28.129315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:28.129287 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:33.068881 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:37:33.068849 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:37:34.829886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:34.829856 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:35.096943 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:35.096916 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:36.254579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:36.254547 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:41.297544 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.297502 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf"] Nov 03 21:37:41.321963 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.321938 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.361256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.361218 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf"] Nov 03 21:37:41.409074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.409033 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-29472d-serving-cert\"" Nov 03 21:37:41.426642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.426585 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafa7e11-4065-4d67-b27b-92697f0023af-openshift-service-ca\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.426974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.426951 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/cafa7e11-4065-4d67-b27b-92697f0023af-kube-api-access-f2kkk\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.427155 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.427136 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-metrics-certs\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.427318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.427301 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.530962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.530928 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-metrics-certs\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.531246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.531223 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.531489 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.531469 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafa7e11-4065-4d67-b27b-92697f0023af-openshift-service-ca\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.531733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.531714 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/cafa7e11-4065-4d67-b27b-92697f0023af-kube-api-access-f2kkk\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.532654 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:37:41.532634 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-serving-cert: secret "maas-api-e2e-api-a-29472d-serving-cert" not found Nov 03 21:37:41.532860 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:37:41.532844 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls podName:cafa7e11-4065-4d67-b27b-92697f0023af nodeName:}" failed. No retries permitted until 2026-11-03 21:37:42.03282244 +0000 UTC m=+1329.643670252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls") pod "maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" (UID: "cafa7e11-4065-4d67-b27b-92697f0023af") : secret "maas-api-e2e-api-a-29472d-serving-cert" not found Nov 03 21:37:41.535481 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.535460 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafa7e11-4065-4d67-b27b-92697f0023af-openshift-service-ca\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.538450 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.538428 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-metrics-certs\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.599259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.599230 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/cafa7e11-4065-4d67-b27b-92697f0023af-kube-api-access-f2kkk\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:41.600761 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.600733 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:41.841514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:41.841484 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:42.040419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:42.040334 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:42.042803 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:42.042782 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls\") pod \"maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:42.080197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:42.080171 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:42.267884 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:42.267858 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:42.646981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:42.646935 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf"] Nov 03 21:37:42.649016 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:37:42.648991 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafa7e11_4065_4d67_b27b_92697f0023af.slice/crio-0ae0134536443e71c5454be5e15b3536017444f0ae7cebbf34534f991eea564a WatchSource:0}: Error finding container 0ae0134536443e71c5454be5e15b3536017444f0ae7cebbf34534f991eea564a: Status 404 returned error can't find the container with id 0ae0134536443e71c5454be5e15b3536017444f0ae7cebbf34534f991eea564a Nov 03 21:37:43.109457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:43.109430 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:43.605262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:43.605232 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" event={"ID":"cafa7e11-4065-4d67-b27b-92697f0023af","Type":"ContainerStarted","Data":"44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5"} Nov 03 21:37:43.605388 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:43.605290 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" event={"ID":"cafa7e11-4065-4d67-b27b-92697f0023af","Type":"ContainerStarted","Data":"0ae0134536443e71c5454be5e15b3536017444f0ae7cebbf34534f991eea564a"} Nov 03 21:37:43.605388 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:43.605320 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:43.730410 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:43.730351 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" podStartSLOduration=1.884053354 podStartE2EDuration="2.730333679s" totalImagesPullingTime="846.280325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:43.724831928 +0000 UTC m=+1331.335679746" watchObservedRunningTime="2026-11-03 21:37:43.730333679 +0000 UTC m=+1331.341181495" Nov 03 21:37:46.846990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:46.846959 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:49.617220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:49.617194 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:37:51.183568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.183537 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc"] Nov 03 21:37:51.187858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.187838 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.240814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.240793 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c5025c-serving-cert\"" Nov 03 21:37:51.251433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.251395 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-metrics-certs\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.251540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.251464 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.251696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.251671 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f243fff-cdba-45cd-8096-397653259359-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.251770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.251735 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ddf\" (UniqueName: \"kubernetes.io/projected/5f243fff-cdba-45cd-8096-397653259359-kube-api-access-v4ddf\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.293453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.293431 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc"] Nov 03 21:37:51.352482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.352456 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-metrics-certs\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.352604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.352548 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.352700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.352646 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f243fff-cdba-45cd-8096-397653259359-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.352700 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:37:51.352681 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-serving-cert: secret "maas-api-e2e-api-a-c5025c-serving-cert" not found Nov 03 21:37:51.352789 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.352704 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ddf\" (UniqueName: \"kubernetes.io/projected/5f243fff-cdba-45cd-8096-397653259359-kube-api-access-v4ddf\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.352789 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:37:51.352747 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls podName:5f243fff-cdba-45cd-8096-397653259359 nodeName:}" failed. No retries permitted until 2026-11-03 21:37:51.852729291 +0000 UTC m=+1339.463577085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls") pod "maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" (UID: "5f243fff-cdba-45cd-8096-397653259359") : secret "maas-api-e2e-api-a-c5025c-serving-cert" not found Nov 03 21:37:51.353345 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.353328 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f243fff-cdba-45cd-8096-397653259359-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.354962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.354945 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-metrics-certs\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.425205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.425184 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ddf\" (UniqueName: \"kubernetes.io/projected/5f243fff-cdba-45cd-8096-397653259359-kube-api-access-v4ddf\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.520965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.520879 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:51.757290 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.757254 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:51.859457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.859429 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:51.861847 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:51.861828 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls\") pod \"maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:52.100829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:52.100800 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:52.489126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:52.489104 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc"] Nov 03 21:37:52.491692 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:37:52.491660 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f243fff_cdba_45cd_8096_397653259359.slice/crio-26e96a4fc8a3cb74a3fa9f12c492b240c223ade89af576d932fb046a1f177f3e WatchSource:0}: Error finding container 26e96a4fc8a3cb74a3fa9f12c492b240c223ade89af576d932fb046a1f177f3e: Status 404 returned error can't find the container with id 26e96a4fc8a3cb74a3fa9f12c492b240c223ade89af576d932fb046a1f177f3e Nov 03 21:37:52.541688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:52.541663 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:52.666302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:52.666274 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" event={"ID":"5f243fff-cdba-45cd-8096-397653259359","Type":"ContainerStarted","Data":"26e96a4fc8a3cb74a3fa9f12c492b240c223ade89af576d932fb046a1f177f3e"} Nov 03 21:37:53.674775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:53.674748 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" event={"ID":"5f243fff-cdba-45cd-8096-397653259359","Type":"ContainerStarted","Data":"44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98"} Nov 03 21:37:53.675150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:53.674829 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:37:53.790867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:53.790791 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" podStartSLOduration=2.013132651 podStartE2EDuration="2.790774359s" totalImagesPullingTime="777.641708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:53.789701735 +0000 UTC m=+1341.400549550" watchObservedRunningTime="2026-11-03 21:37:53.790774359 +0000 UTC m=+1341.401622174" Nov 03 21:37:56.423555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:56.423515 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:59.687026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:37:59.687000 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:38:01.148510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:01.148469 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:02.053211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:02.053177 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:02.699901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:02.699859 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:03.069675 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:38:03.069556 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:38:03.264845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:03.264813 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:06.057422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:06.057377 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:06.750982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:06.750949 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:07.074526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:07.074449 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:09.960800 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:09.960765 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:10.185881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.185841 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:10.276101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.276002 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp"] Nov 03 21:38:10.286109 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.286076 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.336238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.336205 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp"] Nov 03 21:38:10.345327 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.343519 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7896m\" (UniqueName: \"kubernetes.io/projected/cddceccb-7d62-43b5-abad-da056ea7fc0c-kube-api-access-7896m\") pod \"payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.345327 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.343724 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cddceccb-7d62-43b5-abad-da056ea7fc0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.380257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.380221 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-30f143\"" Nov 03 21:38:10.380416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.380302 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-30f143-dockercfg-6nqqd\"" Nov 03 21:38:10.450678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.449748 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7896m\" (UniqueName: \"kubernetes.io/projected/cddceccb-7d62-43b5-abad-da056ea7fc0c-kube-api-access-7896m\") pod \"payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.450678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.449871 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cddceccb-7d62-43b5-abad-da056ea7fc0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.450930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.450907 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cddceccb-7d62-43b5-abad-da056ea7fc0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.507948 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.507913 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7896m\" (UniqueName: \"kubernetes.io/projected/cddceccb-7d62-43b5-abad-da056ea7fc0c-kube-api-access-7896m\") pod \"payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.610524 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.610497 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:10.991073 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:10.991050 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp"] Nov 03 21:38:10.991929 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:38:10.991903 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddceccb_7d62_43b5_abad_da056ea7fc0c.slice/crio-9dc603b02d0659077182dc0171f89e7b8ca72a1f4186eddc29535e9dfcf22ea4 WatchSource:0}: Error finding container 9dc603b02d0659077182dc0171f89e7b8ca72a1f4186eddc29535e9dfcf22ea4: Status 404 returned error can't find the container with id 9dc603b02d0659077182dc0171f89e7b8ca72a1f4186eddc29535e9dfcf22ea4 Nov 03 21:38:11.399650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:11.399583 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:11.790881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:11.790805 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" event={"ID":"cddceccb-7d62-43b5-abad-da056ea7fc0c","Type":"ContainerStarted","Data":"95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f"} Nov 03 21:38:11.790881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:11.790855 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" event={"ID":"cddceccb-7d62-43b5-abad-da056ea7fc0c","Type":"ContainerStarted","Data":"9dc603b02d0659077182dc0171f89e7b8ca72a1f4186eddc29535e9dfcf22ea4"} Nov 03 21:38:11.791060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:11.791003 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:11.905770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:11.905725 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" podStartSLOduration=1.9057135779999999 podStartE2EDuration="1.905713578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:11.902148161 +0000 UTC m=+1359.512995976" watchObservedRunningTime="2026-11-03 21:38:11.905713578 +0000 UTC m=+1359.516561392" Nov 03 21:38:16.580411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:16.580375 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:21.348318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:21.348290 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:22.800226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:22.800198 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:38:26.166106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:26.166059 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:31.085328 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:31.085289 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:33.068486 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:38:33.068447 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:38:34.844204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:34.844173 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:34.993721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:34.993688 5931 generic.go:356] "Generic (PLEG): container finished" podID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerID="8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271" exitCode=0 Nov 03 21:38:34.993861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:34.993762 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" event={"ID":"6d4287f3-df53-47db-874c-3e8d3bfd92f6","Type":"ContainerDied","Data":"8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271"} Nov 03 21:38:34.994144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:34.994130 5931 scope.go:122] "RemoveContainer" containerID="8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271" Nov 03 21:38:34.995537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:34.995510 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:38:34.995593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:34.995580 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:38:34.995653 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:34.995630 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:38:35.656087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:35.656053 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:36.003000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:36.002914 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" event={"ID":"6d4287f3-df53-47db-874c-3e8d3bfd92f6","Type":"ContainerStarted","Data":"49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3"} Nov 03 21:38:36.004376 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:36.004358 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:38:36.005526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:36.005505 5931 patch_prober.go:29] interesting pod/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 21:38:36.005606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:36.005544 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 21:38:36.149590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:36.149377 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:36.371435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:36.371392 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:36.703496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:36.703423 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:37.004864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:37.004749 5931 patch_prober.go:29] interesting pod/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 21:38:37.004864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:37.004833 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 21:38:38.004825 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:38.004795 5931 patch_prober.go:29] interesting pod/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 21:38:38.005033 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:38.004855 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 21:38:38.256240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:38.256167 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:38:39.005040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:39.005008 5931 patch_prober.go:29] interesting pod/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 21:38:39.005427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:39.005068 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 21:38:40.004677 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:40.004594 5931 patch_prober.go:29] interesting pod/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 21:38:40.004880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:40.004688 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 21:38:41.004811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:41.004779 5931 patch_prober.go:29] interesting pod/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 21:38:41.005379 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:41.005346 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 21:38:41.053453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:41.053422 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:41.825111 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:41.825073 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:42.021447 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:42.021410 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:38:42.022802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:42.022778 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:38:42.178072 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:42.178037 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:42.811516 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:42.811482 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:43.298717 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.298681 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n"] Nov 03 21:38:43.302229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.302207 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.332439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332418 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-certs\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.332565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332444 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-credential-socket\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.332565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332469 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754a057d-4185-4c25-aef2-8f2f0f0d335b-istiod-ca-cert\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.332725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332705 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-data\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.332837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332821 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-socket\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.332888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332857 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-envoy\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.332939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332912 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2g9\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-kube-api-access-ph2g9\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.333013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.332994 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-podinfo\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.333067 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.333030 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-token\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.356148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.356123 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n"] Nov 03 21:38:43.401748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.401720 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dad161-openshift-default-dockercfg-pz957\"" Nov 03 21:38:43.442518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.442481 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2g9\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-kube-api-access-ph2g9\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.442762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.442548 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-podinfo\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.442762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.442579 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-token\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.442762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.442673 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-certs\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.442762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.442697 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-credential-socket\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.442762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.442727 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754a057d-4185-4c25-aef2-8f2f0f0d335b-istiod-ca-cert\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.443051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.442825 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-data\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.443510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.443485 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-socket\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.443649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.443526 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-envoy\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.450993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.448819 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754a057d-4185-4c25-aef2-8f2f0f0d335b-istiod-ca-cert\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.450993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.449150 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-credential-socket\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.450993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.449385 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-data\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.450993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.449604 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-socket\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.455594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.454650 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-podinfo\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.479668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.475885 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-certs\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.483666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.482221 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:43.483666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.483228 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-envoy\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.510871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.510843 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-token\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.531981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.531948 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2g9\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-kube-api-access-ph2g9\") pod \"e2e-api-b-dad161-openshift-default-7476c569-xbg8n\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.616205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.616172 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:43.761052 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:43.761015 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:44.058054 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:44.057987 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n"] Nov 03 21:38:44.099710 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:38:44.099686 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754a057d_4185_4c25_aef2_8f2f0f0d335b.slice/crio-cf2b67f2e6e8dc79d6f3d1c7ec41519ca41fefecb19fcbd047a4c0bdfac902b5 WatchSource:0}: Error finding container cf2b67f2e6e8dc79d6f3d1c7ec41519ca41fefecb19fcbd047a4c0bdfac902b5: Status 404 returned error can't find the container with id cf2b67f2e6e8dc79d6f3d1c7ec41519ca41fefecb19fcbd047a4c0bdfac902b5 Nov 03 21:38:44.102100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:44.102071 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:38:44.102179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:44.102134 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:38:44.102179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:44.102162 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:38:45.102365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:45.102323 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" event={"ID":"754a057d-4185-4c25-aef2-8f2f0f0d335b","Type":"ContainerStarted","Data":"c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb"} Nov 03 21:38:45.102790 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:45.102416 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" event={"ID":"754a057d-4185-4c25-aef2-8f2f0f0d335b","Type":"ContainerStarted","Data":"cf2b67f2e6e8dc79d6f3d1c7ec41519ca41fefecb19fcbd047a4c0bdfac902b5"} Nov 03 21:38:45.270484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:45.270440 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podStartSLOduration=2.270427787 podStartE2EDuration="2.270427787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:45.266631158 +0000 UTC m=+1392.877478974" watchObservedRunningTime="2026-11-03 21:38:45.270427787 +0000 UTC m=+1392.881275602" Nov 03 21:38:45.565508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:45.565435 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:45.616499 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:45.616469 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:45.621528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:45.621507 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:46.109205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:46.109178 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:46.109993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:46.109973 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:38:46.233264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:46.233236 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:46.516247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:46.516097 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:46.753633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:46.751856 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:48.172569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:48.172537 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:48.417142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:48.417113 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:51.097461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.097423 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt"] Nov 03 21:38:51.101749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.101726 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.145326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.145290 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.145467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.145396 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-metrics-certs\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.145558 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.145541 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b876b70b-abc3-49fa-b206-70bc6ccdfc76-openshift-service-ca\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.145806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.145786 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4qd9\" (UniqueName: \"kubernetes.io/projected/b876b70b-abc3-49fa-b206-70bc6ccdfc76-kube-api-access-j4qd9\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.203319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.203288 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-30f143-serving-cert\"" Nov 03 21:38:51.206708 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.206681 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt"] Nov 03 21:38:51.247756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.247721 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4qd9\" (UniqueName: \"kubernetes.io/projected/b876b70b-abc3-49fa-b206-70bc6ccdfc76-kube-api-access-j4qd9\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.247908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.247786 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.247908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.247843 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-metrics-certs\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.248024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.247950 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b876b70b-abc3-49fa-b206-70bc6ccdfc76-openshift-service-ca\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.248921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.248896 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b876b70b-abc3-49fa-b206-70bc6ccdfc76-openshift-service-ca\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.249279 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:38:51.249253 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-serving-cert: secret "maas-api-e2e-api-a-30f143-serving-cert" not found Nov 03 21:38:51.249355 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:38:51.249329 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls podName:b876b70b-abc3-49fa-b206-70bc6ccdfc76 nodeName:}" failed. No retries permitted until 2026-11-03 21:38:51.7493097 +0000 UTC m=+1399.360157500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls") pod "maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" (UID: "b876b70b-abc3-49fa-b206-70bc6ccdfc76") : secret "maas-api-e2e-api-a-30f143-serving-cert" not found Nov 03 21:38:51.252792 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.252770 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-metrics-certs\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.317123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.317064 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4qd9\" (UniqueName: \"kubernetes.io/projected/b876b70b-abc3-49fa-b206-70bc6ccdfc76-kube-api-access-j4qd9\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.382447 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.382377 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:51.610269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.610238 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:51.756683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.755931 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.759368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.759339 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls\") pod \"maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:51.760334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.760306 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt"] Nov 03 21:38:51.764897 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.764875 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:51.817806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.817766 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dad161-dockercfg-t5mnn\"" Nov 03 21:38:51.817994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.817880 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dad161\"" Nov 03 21:38:51.821216 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.821191 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt"] Nov 03 21:38:51.847716 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.842328 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:51.866000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.865523 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3189da-e24c-4663-90fb-2a934bda2de4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:51.866000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.865777 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8f5\" (UniqueName: \"kubernetes.io/projected/1f3189da-e24c-4663-90fb-2a934bda2de4-kube-api-access-kg8f5\") pod \"payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:51.975678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.967873 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3189da-e24c-4663-90fb-2a934bda2de4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:51.975678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.968575 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8f5\" (UniqueName: \"kubernetes.io/projected/1f3189da-e24c-4663-90fb-2a934bda2de4-kube-api-access-kg8f5\") pod \"payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:51.975678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:51.970579 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3189da-e24c-4663-90fb-2a934bda2de4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:52.020713 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:52.019842 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:52.026879 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:52.026846 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8f5\" (UniqueName: \"kubernetes.io/projected/1f3189da-e24c-4663-90fb-2a934bda2de4-kube-api-access-kg8f5\") pod \"payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:52.098630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:52.097436 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:52.416788 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:52.416761 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt"] Nov 03 21:38:52.417791 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:38:52.417764 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb876b70b_abc3_49fa_b206_70bc6ccdfc76.slice/crio-008e737bff6252e356b4f2f96ba76c1b4446b4dbe8cfc852949ed63fca7ac123 WatchSource:0}: Error finding container 008e737bff6252e356b4f2f96ba76c1b4446b4dbe8cfc852949ed63fca7ac123: Status 404 returned error can't find the container with id 008e737bff6252e356b4f2f96ba76c1b4446b4dbe8cfc852949ed63fca7ac123 Nov 03 21:38:52.530636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:52.529076 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt"] Nov 03 21:38:52.537741 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:38:52.537717 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3189da_e24c_4663_90fb_2a934bda2de4.slice/crio-b1aaf5c9f0e48ebc6ccd666b06a74a05a83a7cc3e6375f5b57fe895bdf2fe548 WatchSource:0}: Error finding container b1aaf5c9f0e48ebc6ccd666b06a74a05a83a7cc3e6375f5b57fe895bdf2fe548: Status 404 returned error can't find the container with id b1aaf5c9f0e48ebc6ccd666b06a74a05a83a7cc3e6375f5b57fe895bdf2fe548 Nov 03 21:38:53.161922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:53.161892 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" event={"ID":"b876b70b-abc3-49fa-b206-70bc6ccdfc76","Type":"ContainerStarted","Data":"008e737bff6252e356b4f2f96ba76c1b4446b4dbe8cfc852949ed63fca7ac123"} Nov 03 21:38:53.163455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:53.163432 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" event={"ID":"1f3189da-e24c-4663-90fb-2a934bda2de4","Type":"ContainerStarted","Data":"6a5a4a0d0b15cb81e27186401ef43236f6a2a3be8c727d9b0f051e77adc3e24d"} Nov 03 21:38:53.163561 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:53.163473 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" event={"ID":"1f3189da-e24c-4663-90fb-2a934bda2de4","Type":"ContainerStarted","Data":"b1aaf5c9f0e48ebc6ccd666b06a74a05a83a7cc3e6375f5b57fe895bdf2fe548"} Nov 03 21:38:53.163561 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:53.163510 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:38:53.277779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:53.277642 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" podStartSLOduration=2.277623927 podStartE2EDuration="2.277623927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:53.276064591 +0000 UTC m=+1400.886912406" watchObservedRunningTime="2026-11-03 21:38:53.277623927 +0000 UTC m=+1400.888471735" Nov 03 21:38:54.171361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:54.171316 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" event={"ID":"b876b70b-abc3-49fa-b206-70bc6ccdfc76","Type":"ContainerStarted","Data":"726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5"} Nov 03 21:38:54.171823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:54.171381 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:38:54.285687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:54.285606 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" podStartSLOduration=2.376725019 podStartE2EDuration="3.285591748s" totalImagesPullingTime="908.866729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:54.283491643 +0000 UTC m=+1401.894339457" watchObservedRunningTime="2026-11-03 21:38:54.285591748 +0000 UTC m=+1401.896439561" Nov 03 21:38:57.030542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:57.030514 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:57.672966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:57.672899 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:58.582488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:58.582421 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:59.216125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:59.216070 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:59.748965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:38:59.748935 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:00.182539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:00.182517 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:39:02.246365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:02.246329 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:03.067502 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:39:03.067465 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:39:03.850754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:03.850724 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:04.157749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:04.157677 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:04.174420 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:04.174401 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:39:07.138405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.138360 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:07.404728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.404627 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:07.581795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.581752 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f"] Nov 03 21:39:07.595924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.595894 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:07.647665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.646184 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:07.681417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.675010 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f"] Nov 03 21:39:07.698097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.696839 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fff189\"" Nov 03 21:39:07.698097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.697033 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fff189-dockercfg-qq7zs\"" Nov 03 21:39:07.721271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.721235 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664373db-8da4-482d-b5e3-70fad0c56bf8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:07.726515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.726486 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcr5n\" (UniqueName: \"kubernetes.io/projected/664373db-8da4-482d-b5e3-70fad0c56bf8-kube-api-access-xcr5n\") pod \"payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:07.833580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.833536 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664373db-8da4-482d-b5e3-70fad0c56bf8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:07.833958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.833928 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcr5n\" (UniqueName: \"kubernetes.io/projected/664373db-8da4-482d-b5e3-70fad0c56bf8-kube-api-access-xcr5n\") pod \"payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:07.834405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.834386 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664373db-8da4-482d-b5e3-70fad0c56bf8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:07.891346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.891317 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcr5n\" (UniqueName: \"kubernetes.io/projected/664373db-8da4-482d-b5e3-70fad0c56bf8-kube-api-access-xcr5n\") pod \"payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:07.956245 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:07.956187 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:08.331507 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:08.331481 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f"] Nov 03 21:39:08.332511 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:39:08.332487 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664373db_8da4_482d_b5e3_70fad0c56bf8.slice/crio-bd0b4004b86a3385d22f0e78cd3a1c26c93da6ad46654a59b21158910c09ab2d WatchSource:0}: Error finding container bd0b4004b86a3385d22f0e78cd3a1c26c93da6ad46654a59b21158910c09ab2d: Status 404 returned error can't find the container with id bd0b4004b86a3385d22f0e78cd3a1c26c93da6ad46654a59b21158910c09ab2d Nov 03 21:39:09.290674 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:09.290643 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" event={"ID":"664373db-8da4-482d-b5e3-70fad0c56bf8","Type":"ContainerStarted","Data":"f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198"} Nov 03 21:39:09.290861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:09.290695 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" event={"ID":"664373db-8da4-482d-b5e3-70fad0c56bf8","Type":"ContainerStarted","Data":"bd0b4004b86a3385d22f0e78cd3a1c26c93da6ad46654a59b21158910c09ab2d"} Nov 03 21:39:09.290861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:09.290839 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:09.408580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:09.408531 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" podStartSLOduration=2.408519795 podStartE2EDuration="2.408519795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:09.405165795 +0000 UTC m=+1417.016013605" watchObservedRunningTime="2026-11-03 21:39:09.408519795 +0000 UTC m=+1417.019367609" Nov 03 21:39:12.643929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:12.643881 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:13.350664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:13.350587 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:13.578869 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:13.578816 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:17.432763 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:17.432729 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:20.300295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:20.300268 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:39:22.151119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:22.151083 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:26.821627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:26.821561 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:29.102419 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:29.102383 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:31.940895 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:31.940865 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:33.070482 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:39:33.070445 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:39:36.850858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:36.850825 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:41.276927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.276892 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv"] Nov 03 21:39:41.282258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.282239 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.333738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.333684 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dad161-serving-cert\"" Nov 03 21:39:41.388730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.388705 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv"] Nov 03 21:39:41.439693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.439663 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34277911-825b-4fb2-8fe2-2bea9eeadbc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.439849 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.439791 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.439849 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.439812 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzbx\" (UniqueName: \"kubernetes.io/projected/34277911-825b-4fb2-8fe2-2bea9eeadbc1-kube-api-access-gbzbx\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.440112 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.440084 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-metrics-certs\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.544958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.544868 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34277911-825b-4fb2-8fe2-2bea9eeadbc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.545178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.545032 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.545178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.545060 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzbx\" (UniqueName: \"kubernetes.io/projected/34277911-825b-4fb2-8fe2-2bea9eeadbc1-kube-api-access-gbzbx\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.545290 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:39:41.545202 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-serving-cert: secret "maas-api-e2e-api-b-dad161-serving-cert" not found Nov 03 21:39:41.545290 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:39:41.545288 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls podName:34277911-825b-4fb2-8fe2-2bea9eeadbc1 nodeName:}" failed. No retries permitted until 2026-11-03 21:39:42.045267295 +0000 UTC m=+1449.656115108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls") pod "maas-api-e2e-api-b-dad161-757d859954-r59jv" (UID: "34277911-825b-4fb2-8fe2-2bea9eeadbc1") : secret "maas-api-e2e-api-b-dad161-serving-cert" not found Nov 03 21:39:41.545383 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.545365 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-metrics-certs\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.545835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.545812 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34277911-825b-4fb2-8fe2-2bea9eeadbc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.552191 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.552170 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-metrics-certs\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.602746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.602707 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzbx\" (UniqueName: \"kubernetes.io/projected/34277911-825b-4fb2-8fe2-2bea9eeadbc1-kube-api-access-gbzbx\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:41.661726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.661693 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:41.895040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:41.895005 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:42.053744 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:42.053715 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:42.056073 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:42.056051 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls\") pod \"maas-api-e2e-api-b-dad161-757d859954-r59jv\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:42.195082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:42.195028 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:42.569114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:42.569094 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv"] Nov 03 21:39:42.571080 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:39:42.571054 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34277911_825b_4fb2_8fe2_2bea9eeadbc1.slice/crio-a35de3ca34f22ea7811b0eca50cd937634c0931ae7b8289761ad2aa9518caa33 WatchSource:0}: Error finding container a35de3ca34f22ea7811b0eca50cd937634c0931ae7b8289761ad2aa9518caa33: Status 404 returned error can't find the container with id a35de3ca34f22ea7811b0eca50cd937634c0931ae7b8289761ad2aa9518caa33 Nov 03 21:39:43.579122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:43.579090 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" event={"ID":"34277911-825b-4fb2-8fe2-2bea9eeadbc1","Type":"ContainerStarted","Data":"0b25ddff6836f111941ad7150c3894879cbf3bdf48d3b3d3501402683738f180"} Nov 03 21:39:43.579490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:43.579141 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" event={"ID":"34277911-825b-4fb2-8fe2-2bea9eeadbc1","Type":"ContainerStarted","Data":"a35de3ca34f22ea7811b0eca50cd937634c0931ae7b8289761ad2aa9518caa33"} Nov 03 21:39:43.579490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:43.579171 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:43.692987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:43.692901 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" podStartSLOduration=1.871232429 podStartE2EDuration="2.692887812s" totalImagesPullingTime="821.655383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:43.691845351 +0000 UTC m=+1451.302693167" watchObservedRunningTime="2026-11-03 21:39:43.692887812 +0000 UTC m=+1451.303735626" Nov 03 21:39:43.797785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:43.797755 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:46.540717 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:46.540683 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:49.592602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:49.592565 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:39:51.619579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:51.619543 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:54.104126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:54.104092 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:54.844775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:54.844746 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:55.496783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:55.496748 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:55.810350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:55.810271 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:56.903393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:56.903357 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:58.632703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:58.632672 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:58.863409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:58.863379 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:59.443574 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:59.443527 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:59.687622 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:39:59.687575 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:02.127811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:02.127771 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:02.357977 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:02.357945 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:03.068311 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:40:03.068278 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:40:04.434007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.433963 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:04.665362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.665329 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:04.718040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.717962 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q"] Nov 03 21:40:04.722392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.722363 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:04.780070 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.780039 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q"] Nov 03 21:40:04.812572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.812537 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b44aee0-da1d-4e4e-a775-0842abb2ae51-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:04.813026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.812995 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jbhh\" (UniqueName: \"kubernetes.io/projected/7b44aee0-da1d-4e4e-a775-0842abb2ae51-kube-api-access-2jbhh\") pod \"payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:04.824198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.824168 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-85d7ff\"" Nov 03 21:40:04.826111 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.826089 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-85d7ff-dockercfg-drtk5\"" Nov 03 21:40:04.879557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.879523 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59"] Nov 03 21:40:04.884374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.884352 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:04.915050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.915016 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jbhh\" (UniqueName: \"kubernetes.io/projected/7b44aee0-da1d-4e4e-a775-0842abb2ae51-kube-api-access-2jbhh\") pod \"payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:04.915274 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.915242 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b44aee0-da1d-4e4e-a775-0842abb2ae51-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:04.916257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.916224 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b44aee0-da1d-4e4e-a775-0842abb2ae51-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:04.930641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.930582 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59"] Nov 03 21:40:04.977526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:04.977452 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jbhh\" (UniqueName: \"kubernetes.io/projected/7b44aee0-da1d-4e4e-a775-0842abb2ae51-kube-api-access-2jbhh\") pod \"payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:05.017181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.017153 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttd5k\" (UniqueName: \"kubernetes.io/projected/4ca62afe-8252-49db-99f9-d591f81b4392-kube-api-access-ttd5k\") pod \"payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.017395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.017299 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca62afe-8252-49db-99f9-d591f81b4392-plugins-config-volume\") pod \"payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.042595 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.042568 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:05.119452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.119418 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca62afe-8252-49db-99f9-d591f81b4392-plugins-config-volume\") pod \"payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.119714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.119696 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttd5k\" (UniqueName: \"kubernetes.io/projected/4ca62afe-8252-49db-99f9-d591f81b4392-kube-api-access-ttd5k\") pod \"payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.120159 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.120141 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca62afe-8252-49db-99f9-d591f81b4392-plugins-config-volume\") pod \"payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.194813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.185508 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttd5k\" (UniqueName: \"kubernetes.io/projected/4ca62afe-8252-49db-99f9-d591f81b4392-kube-api-access-ttd5k\") pod \"payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.204940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.204907 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.463225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.463200 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q"] Nov 03 21:40:05.464672 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:40:05.464642 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b44aee0_da1d_4e4e_a775_0842abb2ae51.slice/crio-a866536d493b5a3aa07c6a99e1f0d698f780684961d9edaaf12546b8648a56b7 WatchSource:0}: Error finding container a866536d493b5a3aa07c6a99e1f0d698f780684961d9edaaf12546b8648a56b7: Status 404 returned error can't find the container with id a866536d493b5a3aa07c6a99e1f0d698f780684961d9edaaf12546b8648a56b7 Nov 03 21:40:05.631657 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.631631 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59"] Nov 03 21:40:05.633378 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:40:05.633356 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca62afe_8252_49db_99f9_d591f81b4392.slice/crio-dc20e30906953514ec77ee5c93ad78b5d48c12e0714f2019bc8e4588bc79cba1 WatchSource:0}: Error finding container dc20e30906953514ec77ee5c93ad78b5d48c12e0714f2019bc8e4588bc79cba1: Status 404 returned error can't find the container with id dc20e30906953514ec77ee5c93ad78b5d48c12e0714f2019bc8e4588bc79cba1 Nov 03 21:40:05.787673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.787579 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" event={"ID":"4ca62afe-8252-49db-99f9-d591f81b4392","Type":"ContainerStarted","Data":"a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe"} Nov 03 21:40:05.787784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.787670 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" event={"ID":"4ca62afe-8252-49db-99f9-d591f81b4392","Type":"ContainerStarted","Data":"dc20e30906953514ec77ee5c93ad78b5d48c12e0714f2019bc8e4588bc79cba1"} Nov 03 21:40:05.787784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.787700 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:05.789237 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.789215 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" event={"ID":"7b44aee0-da1d-4e4e-a775-0842abb2ae51","Type":"ContainerStarted","Data":"51f634f8ae36f4ab7f38d24ca1ccffd1f558f825c67d6291079a16ae14b25f22"} Nov 03 21:40:05.789314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.789251 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" event={"ID":"7b44aee0-da1d-4e4e-a775-0842abb2ae51","Type":"ContainerStarted","Data":"a866536d493b5a3aa07c6a99e1f0d698f780684961d9edaaf12546b8648a56b7"} Nov 03 21:40:05.789356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.789338 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:05.902953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:05.902913 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" podStartSLOduration=1.90290016 podStartE2EDuration="1.90290016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:05.899995751 +0000 UTC m=+1473.510843565" watchObservedRunningTime="2026-11-03 21:40:05.90290016 +0000 UTC m=+1473.513747975" Nov 03 21:40:06.017928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:06.013437 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" podStartSLOduration=2.01338371 podStartE2EDuration="2.01338371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:06.011122438 +0000 UTC m=+1473.621970265" watchObservedRunningTime="2026-11-03 21:40:06.01338371 +0000 UTC m=+1473.624231526" Nov 03 21:40:07.415898 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:07.415450 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:12.270942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:12.270902 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:16.153080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.153046 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9"] Nov 03 21:40:16.153867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.153835 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" podUID="b0fda6a1-9f90-4049-99c8-e102e2aecbb8" containerName="maas-api" containerID="cri-o://ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed" gracePeriod=30 Nov 03 21:40:16.349635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.348462 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:16.524916 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.524880 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:40:16.675636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.666041 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-metrics-certs\") pod \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " Nov 03 21:40:16.675636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.666274 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-kube-api-access-vhrbz\" (UniqueName: \"kubernetes.io/projected/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-kube-api-access-vhrbz\") pod \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " Nov 03 21:40:16.675636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.670752 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls\") pod \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " Nov 03 21:40:16.675636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.670902 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-openshift-service-ca\") pod \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\" (UID: \"b0fda6a1-9f90-4049-99c8-e102e2aecbb8\") " Nov 03 21:40:16.681108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.679869 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-openshift-service-ca" pod "b0fda6a1-9f90-4049-99c8-e102e2aecbb8" (UID: "b0fda6a1-9f90-4049-99c8-e102e2aecbb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:40:16.727204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.727167 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-metrics-certs" pod "b0fda6a1-9f90-4049-99c8-e102e2aecbb8" (UID: "b0fda6a1-9f90-4049-99c8-e102e2aecbb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:40:16.729202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.729170 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-kube-api-access-vhrbz" pod "b0fda6a1-9f90-4049-99c8-e102e2aecbb8" (UID: "b0fda6a1-9f90-4049-99c8-e102e2aecbb8"). InnerVolumeSpecName "kube-api-access-vhrbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:40:16.729335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.729236 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls" pod "b0fda6a1-9f90-4049-99c8-e102e2aecbb8" (UID: "b0fda6a1-9f90-4049-99c8-e102e2aecbb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:40:16.772803 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.772718 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhrbz\" (UniqueName: \"kubernetes.io/projected/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-kube-api-access-vhrbz\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:40:16.772803 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.772747 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:40:16.772803 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.772757 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:40:16.772803 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.772768 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fda6a1-9f90-4049-99c8-e102e2aecbb8-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:40:16.800265 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.800237 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:40:16.800404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.800319 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:40:16.893156 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.893114 5931 generic.go:356] "Generic (PLEG): container finished" podID="b0fda6a1-9f90-4049-99c8-e102e2aecbb8" containerID="ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed" exitCode=0 Nov 03 21:40:16.893332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.893217 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" Nov 03 21:40:16.893332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.893254 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" event={"ID":"b0fda6a1-9f90-4049-99c8-e102e2aecbb8","Type":"ContainerDied","Data":"ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed"} Nov 03 21:40:16.893332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.893307 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9" event={"ID":"b0fda6a1-9f90-4049-99c8-e102e2aecbb8","Type":"ContainerDied","Data":"d49957b3451c6cf3fdd0e21a04bb4eade8405fab0df6bb619cf837f889106cd8"} Nov 03 21:40:16.893332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.893329 5931 scope.go:122] "RemoveContainer" containerID="ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed" Nov 03 21:40:16.907751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.907729 5931 scope.go:122] "RemoveContainer" containerID="ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed" Nov 03 21:40:16.908485 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:40:16.908464 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed\": container with ID starting with ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed not found: ID does not exist" containerID="ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed" Nov 03 21:40:16.908579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:16.908494 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed"} err="failed to get container status \"ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed\": rpc error: code = NotFound desc = could not find container \"ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed\": container with ID starting with ab753fa7f696335233273940bc179029744a29a47363021b7e837184030dd2ed not found: ID does not exist" Nov 03 21:40:17.142155 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:17.142122 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9"] Nov 03 21:40:17.251635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:17.201427 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-lkpf9"] Nov 03 21:40:17.867501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:17.867470 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:18.126767 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:18.126680 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:18.366787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:18.366744 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:18.509132 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:18.509046 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fda6a1-9f90-4049-99c8-e102e2aecbb8" path="/var/lib/kubelet/pods/b0fda6a1-9f90-4049-99c8-e102e2aecbb8/volumes" Nov 03 21:40:20.926122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:20.926088 5931 generic.go:356] "Generic (PLEG): container finished" podID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerID="c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb" exitCode=0 Nov 03 21:40:20.926524 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:20.926154 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" event={"ID":"754a057d-4185-4c25-aef2-8f2f0f0d335b","Type":"ContainerDied","Data":"c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb"} Nov 03 21:40:20.926524 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:20.926468 5931 scope.go:122] "RemoveContainer" containerID="c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb" Nov 03 21:40:20.927867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:20.927844 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:40:20.927940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:20.927907 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:40:20.927975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:20.927939 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:40:21.955940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:21.955904 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" event={"ID":"754a057d-4185-4c25-aef2-8f2f0f0d335b","Type":"ContainerStarted","Data":"8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3"} Nov 03 21:40:22.554579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:22.554524 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:22.616967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:22.616936 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:40:22.617125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:22.617029 5931 patch_prober.go:29] interesting pod/e2e-api-b-dad161-openshift-default-7476c569-xbg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 21:40:22.617125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:22.617068 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 21:40:23.341634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:23.333086 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:23.617456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:23.617379 5931 patch_prober.go:29] interesting pod/e2e-api-b-dad161-openshift-default-7476c569-xbg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 21:40:23.617456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:23.617437 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 21:40:24.616948 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:24.616916 5931 patch_prober.go:29] interesting pod/e2e-api-b-dad161-openshift-default-7476c569-xbg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 21:40:24.617341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:24.616962 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 21:40:25.617362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:25.617328 5931 patch_prober.go:29] interesting pod/e2e-api-b-dad161-openshift-default-7476c569-xbg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 21:40:25.617890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:25.617859 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 21:40:26.617404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:26.617368 5931 patch_prober.go:29] interesting pod/e2e-api-b-dad161-openshift-default-7476c569-xbg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 21:40:26.617846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:26.617426 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 21:40:27.621693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:27.621670 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:40:27.622115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:27.622097 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:40:27.622831 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:27.622811 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:40:28.272540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:28.272509 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:29.417996 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:29.417929 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:29.650088 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:29.650050 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:33.068279 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:40:33.068240 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:40:33.503237 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:33.503204 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:34.200920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.194592 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:34.420231 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.420195 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:34.460359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.460278 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l"] Nov 03 21:40:34.461069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.461046 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b0fda6a1-9f90-4049-99c8-e102e2aecbb8" containerName="maas-api" Nov 03 21:40:34.461069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.461070 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fda6a1-9f90-4049-99c8-e102e2aecbb8" containerName="maas-api" Nov 03 21:40:34.461273 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.461260 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fda6a1-9f90-4049-99c8-e102e2aecbb8" containerName="maas-api" Nov 03 21:40:34.466214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.466195 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.520562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.520534 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l"] Nov 03 21:40:34.550219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.550185 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.550412 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.550354 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgdf\" (UniqueName: \"kubernetes.io/projected/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-kube-api-access-hdgdf\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.564543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.564518 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c1662df5\"" Nov 03 21:40:34.564766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.564641 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c1662df5-dockercfg-pb4wn\"" Nov 03 21:40:34.620451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.620419 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn"] Nov 03 21:40:34.626840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.626814 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:34.654696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.654652 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.655067 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.655042 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgdf\" (UniqueName: \"kubernetes.io/projected/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-kube-api-access-hdgdf\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.665928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.658911 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.678182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.678148 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn"] Nov 03 21:40:34.716583 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.716401 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgdf\" (UniqueName: \"kubernetes.io/projected/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-kube-api-access-hdgdf\") pod \"payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.772710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.770091 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:34.772710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.770246 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhmn\" (UniqueName: \"kubernetes.io/projected/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-kube-api-access-wdhmn\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:34.788668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.781893 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:34.872130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.872095 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:34.872277 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.872259 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhmn\" (UniqueName: \"kubernetes.io/projected/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-kube-api-access-wdhmn\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:34.873721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.873691 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:34.929548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.929517 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhmn\" (UniqueName: \"kubernetes.io/projected/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-kube-api-access-wdhmn\") pod \"payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:34.984192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:34.984139 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:35.170846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:35.170822 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l"] Nov 03 21:40:35.172336 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:40:35.172311 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92614c7d_1f19_4a6e_95cb_5ba987bd3bb1.slice/crio-27c8fe00a5dbbe14cd58463d828e02b347924b18f2a0b5b911a58c554b9f0219 WatchSource:0}: Error finding container 27c8fe00a5dbbe14cd58463d828e02b347924b18f2a0b5b911a58c554b9f0219: Status 404 returned error can't find the container with id 27c8fe00a5dbbe14cd58463d828e02b347924b18f2a0b5b911a58c554b9f0219 Nov 03 21:40:35.363447 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:35.363426 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn"] Nov 03 21:40:35.364477 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:40:35.364449 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c8c7fd_bc3a_42d3_807c_f77238e0e732.slice/crio-9196affe963beb055f7e029ea09c17638bd7e5f3b9cb82110df26d844016f647 WatchSource:0}: Error finding container 9196affe963beb055f7e029ea09c17638bd7e5f3b9cb82110df26d844016f647: Status 404 returned error can't find the container with id 9196affe963beb055f7e029ea09c17638bd7e5f3b9cb82110df26d844016f647 Nov 03 21:40:35.476263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:35.476237 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:40:35.481168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:35.481148 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:40:36.056291 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.056258 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" event={"ID":"c9c8c7fd-bc3a-42d3-807c-f77238e0e732","Type":"ContainerStarted","Data":"7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78"} Nov 03 21:40:36.056452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.056336 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:36.056452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.056361 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" event={"ID":"c9c8c7fd-bc3a-42d3-807c-f77238e0e732","Type":"ContainerStarted","Data":"9196affe963beb055f7e029ea09c17638bd7e5f3b9cb82110df26d844016f647"} Nov 03 21:40:36.058190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.058149 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" event={"ID":"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1","Type":"ContainerStarted","Data":"d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce"} Nov 03 21:40:36.058303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.058206 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:36.058303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.058228 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" event={"ID":"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1","Type":"ContainerStarted","Data":"27c8fe00a5dbbe14cd58463d828e02b347924b18f2a0b5b911a58c554b9f0219"} Nov 03 21:40:36.176529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.176480 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" podStartSLOduration=2.176468363 podStartE2EDuration="2.176468363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:36.174550575 +0000 UTC m=+1503.785398403" watchObservedRunningTime="2026-11-03 21:40:36.176468363 +0000 UTC m=+1503.787316178" Nov 03 21:40:36.291126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:36.291085 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" podStartSLOduration=2.291072749 podStartE2EDuration="2.291072749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:36.286850473 +0000 UTC m=+1503.897698289" watchObservedRunningTime="2026-11-03 21:40:36.291072749 +0000 UTC m=+1503.901920564" Nov 03 21:40:38.014280 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:38.014248 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:38.236830 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:38.236798 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:38.707277 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:38.707246 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:38.939299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:38.939267 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:44.080486 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:44.080446 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:44.302347 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:44.302318 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:47.068433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:47.068407 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:40:47.068861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:47.068501 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:40:48.939988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:48.939947 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:50.953940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:50.953910 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:54.119569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:54.119531 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:54.804523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:54.804484 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:55.040286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:55.040228 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:59.271718 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:40:59.271682 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:03.068298 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:41:03.068263 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:41:04.096323 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:04.096286 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:04.332584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:04.332542 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:04.577316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:04.577242 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:06.313606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:06.313570 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp"] Nov 03 21:41:06.314090 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:06.313840 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" containerID="cri-o://83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05" gracePeriod=30 Nov 03 21:41:07.155096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:07.155062 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg"] Nov 03 21:41:07.155297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:07.155248 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" containerID="cri-o://16ea3cbd22fcc8435493a8dae87e2fa77cde77593a91480d647870507fe86d02" gracePeriod=30 Nov 03 21:41:07.204178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:07.204143 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl"] Nov 03 21:41:07.204417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:07.204390 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" containerID="cri-o://9b483b04d488577e24be34b8484569ab2820fbc649f8bcb914b940a977bd5fa7" gracePeriod=30 Nov 03 21:41:09.025174 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:09.025129 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:11.589565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.589547 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:41:11.651557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651529 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-kube-api-access-28mnz\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-kube-api-access-28mnz\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.651735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651660 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-podinfo\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.651735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651690 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-envoy\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.651735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651730 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-certs\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.651951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651847 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-socket\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.651951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651886 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-credential-socket\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.651951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651934 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-data\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.652105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.651985 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-certs" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:11.652163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652081 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-token\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.652163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652142 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-credential-socket" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:11.652264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652150 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3b7639-e452-40d5-ab96-20d46bc7a601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b7639-e452-40d5-ab96-20d46bc7a601-istiod-ca-cert\") pod \"6c3b7639-e452-40d5-ab96-20d46bc7a601\" (UID: \"6c3b7639-e452-40d5-ab96-20d46bc7a601\") " Nov 03 21:41:11.652264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652168 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-data" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:11.652459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652436 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3b7639-e452-40d5-ab96-20d46bc7a601-istiod-ca-cert" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:41:11.652697 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652676 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-socket" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:11.652890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652870 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b7639-e452-40d5-ab96-20d46bc7a601-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.652966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652895 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.652966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652909 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.652966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652921 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.652966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.652935 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.654351 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.654326 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-kube-api-access-28mnz" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "kube-api-access-28mnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:11.654454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.654359 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-token" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:11.654454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.654408 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-envoy" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:11.654543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.654460 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-podinfo" pod "6c3b7639-e452-40d5-ab96-20d46bc7a601" (UID: "6c3b7639-e452-40d5-ab96-20d46bc7a601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:41:11.753647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.753592 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mnz\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-kube-api-access-28mnz\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.753757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.753655 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.753757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.753667 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:11.753757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:11.753676 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b7639-e452-40d5-ab96-20d46bc7a601-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.328297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.328268 5931 generic.go:356] "Generic (PLEG): container finished" podID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerID="9b483b04d488577e24be34b8484569ab2820fbc649f8bcb914b940a977bd5fa7" exitCode=0 Nov 03 21:41:12.328413 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.328354 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" event={"ID":"5244e579-2d83-40a1-9de7-a7f92a5b39a3","Type":"ContainerDied","Data":"9b483b04d488577e24be34b8484569ab2820fbc649f8bcb914b940a977bd5fa7"} Nov 03 21:41:12.330120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.330097 5931 generic.go:356] "Generic (PLEG): container finished" podID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerID="83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05" exitCode=0 Nov 03 21:41:12.330241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.330149 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" Nov 03 21:41:12.330241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.330170 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" event={"ID":"6c3b7639-e452-40d5-ab96-20d46bc7a601","Type":"ContainerDied","Data":"83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05"} Nov 03 21:41:12.330241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.330218 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp" event={"ID":"6c3b7639-e452-40d5-ab96-20d46bc7a601","Type":"ContainerDied","Data":"f3fe7b24d1a5972fe3beda724c52d710f38ab0026376cc6e048b5aa0fecc73d5"} Nov 03 21:41:12.330408 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.330248 5931 scope.go:122] "RemoveContainer" containerID="83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05" Nov 03 21:41:12.332932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.332852 5931 generic.go:356] "Generic (PLEG): container finished" podID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerID="16ea3cbd22fcc8435493a8dae87e2fa77cde77593a91480d647870507fe86d02" exitCode=0 Nov 03 21:41:12.332932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.332899 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" event={"ID":"dfe34529-4554-4a26-b59a-483560b5ba1e","Type":"ContainerDied","Data":"16ea3cbd22fcc8435493a8dae87e2fa77cde77593a91480d647870507fe86d02"} Nov 03 21:41:12.361605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.361559 5931 scope.go:122] "RemoveContainer" containerID="83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05" Nov 03 21:41:12.363727 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:41:12.363699 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05\": container with ID starting with 83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05 not found: ID does not exist" containerID="83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05" Nov 03 21:41:12.363837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.363742 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05"} err="failed to get container status \"83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05\": rpc error: code = NotFound desc = could not find container \"83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05\": container with ID starting with 83ab4e40cdad4472aa5d661f4fb1d7ceec342da22e23ce1f6c954e6f09199b05 not found: ID does not exist" Nov 03 21:41:12.486142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.486120 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:41:12.530789 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.530765 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:41:12.546069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.546048 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp"] Nov 03 21:41:12.565822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.565800 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-credential-socket\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.565941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.565849 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-socket\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.566002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.565918 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-kube-api-access-5f4df\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-kube-api-access-5f4df\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.566002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.565969 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-certs\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566005 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-token\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-token\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.566105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566056 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-credential-socket" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.566207 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566110 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istiod-ca-cert\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566207 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566151 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-socket" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.566207 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566191 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-credential-socket\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566217 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-certs" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.566352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566242 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-socket\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566277 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-envoy\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566301 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-kube-api-access-qfqxq\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-kube-api-access-qfqxq\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566345 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe34529-4554-4a26-b59a-483560b5ba1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe34529-4554-4a26-b59a-483560b5ba1e-istiod-ca-cert\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566371 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-podinfo\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566413 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-podinfo\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566440 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-data\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566422 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-credential-socket" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566490 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-data\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566460 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istiod-ca-cert" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566567 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-socket" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.566602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566589 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-envoy\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.567040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566673 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-certs\") pod \"dfe34529-4554-4a26-b59a-483560b5ba1e\" (UID: \"dfe34529-4554-4a26-b59a-483560b5ba1e\") " Nov 03 21:41:12.567040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566730 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-token\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-token\") pod \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\" (UID: \"5244e579-2d83-40a1-9de7-a7f92a5b39a3\") " Nov 03 21:41:12.567040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566758 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-data" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.567040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.566820 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-data" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.567652 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567628 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567655 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567662 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-certs" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567669 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe34529-4554-4a26-b59a-483560b5ba1e-istiod-ca-cert" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567669 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567703 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567721 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567734 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.567751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567747 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.568321 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.567762 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.569390 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.569364 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-kube-api-access-5f4df" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "kube-api-access-5f4df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:12.569701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.569672 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-podinfo" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:41:12.569933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.569907 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-envoy" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.570163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.570137 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-envoy" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:12.570251 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.570182 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-kube-api-access-qfqxq" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "kube-api-access-qfqxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:12.570316 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.570299 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-token" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:12.570390 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.570376 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-token" pod "dfe34529-4554-4a26-b59a-483560b5ba1e" (UID: "dfe34529-4554-4a26-b59a-483560b5ba1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:12.570565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.570552 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-podinfo" pod "5244e579-2d83-40a1-9de7-a7f92a5b39a3" (UID: "5244e579-2d83-40a1-9de7-a7f92a5b39a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:41:12.596198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.596175 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kcdxp"] Nov 03 21:41:12.668852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668829 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668852 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfqxq\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-kube-api-access-qfqxq\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668862 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe34529-4554-4a26-b59a-483560b5ba1e-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668870 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668878 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668886 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668894 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe34529-4554-4a26-b59a-483560b5ba1e-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668903 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5244e579-2d83-40a1-9de7-a7f92a5b39a3-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668912 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4df\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-kube-api-access-5f4df\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:12.668990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:12.668920 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe34529-4554-4a26-b59a-483560b5ba1e-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:13.342337 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.342311 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" Nov 03 21:41:13.342512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.342346 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg" event={"ID":"dfe34529-4554-4a26-b59a-483560b5ba1e","Type":"ContainerDied","Data":"c73d0bcfe0110da4c61d5acc02f92c895d19e254205f176e945052138d210aaf"} Nov 03 21:41:13.342512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.342411 5931 scope.go:122] "RemoveContainer" containerID="16ea3cbd22fcc8435493a8dae87e2fa77cde77593a91480d647870507fe86d02" Nov 03 21:41:13.344027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.344011 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" Nov 03 21:41:13.344027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.344016 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl" event={"ID":"5244e579-2d83-40a1-9de7-a7f92a5b39a3","Type":"ContainerDied","Data":"c78e272a7da38d2b80e4fd199f140a66123501fbfd2a3bf185ee1aa51acf1c7b"} Nov 03 21:41:13.358797 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.358779 5931 scope.go:122] "RemoveContainer" containerID="9b483b04d488577e24be34b8484569ab2820fbc649f8bcb914b940a977bd5fa7" Nov 03 21:41:13.507268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.507243 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg"] Nov 03 21:41:13.555581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.555552 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-ggxpg"] Nov 03 21:41:13.618342 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.618289 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl"] Nov 03 21:41:13.622883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.622863 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-d94vl"] Nov 03 21:41:13.937114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:13.937019 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:14.507206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:14.507174 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" path="/var/lib/kubelet/pods/5244e579-2d83-40a1-9de7-a7f92a5b39a3/volumes" Nov 03 21:41:14.507807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:14.507790 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" path="/var/lib/kubelet/pods/6c3b7639-e452-40d5-ab96-20d46bc7a601/volumes" Nov 03 21:41:14.508370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:14.508354 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" path="/var/lib/kubelet/pods/dfe34529-4554-4a26-b59a-483560b5ba1e/volumes" Nov 03 21:41:16.987342 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:16.987306 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:17.729857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:17.729813 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:18.452710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.452679 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:18.742082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.741991 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:18.992852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.992781 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z"] Nov 03 21:41:18.993389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993376 5931 cpu_manager.go:414] "containerMap: removing container" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" Nov 03 21:41:18.993389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993390 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" Nov 03 21:41:18.993495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993415 5931 cpu_manager.go:414] "containerMap: removing container" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" Nov 03 21:41:18.993495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993421 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" Nov 03 21:41:18.993495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993437 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" Nov 03 21:41:18.993495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993442 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" Nov 03 21:41:18.993633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993534 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe34529-4554-4a26-b59a-483560b5ba1e" containerName="istio-proxy" Nov 03 21:41:18.993633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993541 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3b7639-e452-40d5-ab96-20d46bc7a601" containerName="istio-proxy" Nov 03 21:41:18.993633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.993554 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5244e579-2d83-40a1-9de7-a7f92a5b39a3" containerName="istio-proxy" Nov 03 21:41:18.998813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:18.998794 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.006882 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.006857 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-metrics-certs\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.006980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.006910 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a114a05-b189-41b7-9735-809e1684cee0-openshift-service-ca\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.007101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.007077 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6n5\" (UniqueName: \"kubernetes.io/projected/8a114a05-b189-41b7-9735-809e1684cee0-kube-api-access-df6n5\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.007193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.007178 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.050325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.050307 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c1662df5-serving-cert\"" Nov 03 21:41:19.103461 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.103435 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z"] Nov 03 21:41:19.107934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.107910 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-metrics-certs\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.108022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.107986 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a114a05-b189-41b7-9735-809e1684cee0-openshift-service-ca\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.108110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.108094 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6n5\" (UniqueName: \"kubernetes.io/projected/8a114a05-b189-41b7-9735-809e1684cee0-kube-api-access-df6n5\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.108195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.108176 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.108371 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:41:19.108350 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-serving-cert: secret "maas-api-e2e-mt-c1662df5-serving-cert" not found Nov 03 21:41:19.108476 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:41:19.108434 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls podName:8a114a05-b189-41b7-9735-809e1684cee0 nodeName:}" failed. No retries permitted until 2026-11-03 21:41:19.60841271 +0000 UTC m=+1547.219260524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls") pod "maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" (UID: "8a114a05-b189-41b7-9735-809e1684cee0") : secret "maas-api-e2e-mt-c1662df5-serving-cert" not found Nov 03 21:41:19.108809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.108793 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a114a05-b189-41b7-9735-809e1684cee0-openshift-service-ca\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.110884 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.110861 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-metrics-certs\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.165976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.165955 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6n5\" (UniqueName: \"kubernetes.io/projected/8a114a05-b189-41b7-9735-809e1684cee0-kube-api-access-df6n5\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.377270 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.377238 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:19.606815 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.606784 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:19.613901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.613873 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.616373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.616351 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls\") pod \"maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:19.835753 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.835677 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:19.914499 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:19.914471 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:20.289798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:20.289770 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z"] Nov 03 21:41:20.291310 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:41:20.291285 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a114a05_b189_41b7_9735_809e1684cee0.slice/crio-afe05772ee29b03d46613484d5edf3a333a79ba20b7532a978d6ec3dc18916aa WatchSource:0}: Error finding container afe05772ee29b03d46613484d5edf3a333a79ba20b7532a978d6ec3dc18916aa: Status 404 returned error can't find the container with id afe05772ee29b03d46613484d5edf3a333a79ba20b7532a978d6ec3dc18916aa Nov 03 21:41:20.292797 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:20.292781 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:41:20.397639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:20.397586 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" event={"ID":"8a114a05-b189-41b7-9735-809e1684cee0","Type":"ContainerStarted","Data":"afe05772ee29b03d46613484d5edf3a333a79ba20b7532a978d6ec3dc18916aa"} Nov 03 21:41:21.405888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:21.405802 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" event={"ID":"8a114a05-b189-41b7-9735-809e1684cee0","Type":"ContainerStarted","Data":"fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90"} Nov 03 21:41:21.406298 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:21.406003 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:21.518953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:21.518888 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" podStartSLOduration=2.661740326 podStartE2EDuration="3.518871815s" totalImagesPullingTime="857.131489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:21.517761765 +0000 UTC m=+1549.128609579" watchObservedRunningTime="2026-11-03 21:41:21.518871815 +0000 UTC m=+1549.129719632" Nov 03 21:41:22.249410 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:22.249384 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792"] Nov 03 21:41:22.249590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:22.249555 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" containerID="cri-o://da8587b75d8e48c50d89966c2fdc7257921b3ff5fccb3a68e008c86509a5bd92" gracePeriod=30 Nov 03 21:41:22.486877 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:22.486842 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:22.726671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:22.726643 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:24.773657 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:24.773626 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:27.427023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.426369 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:41:27.458305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.458271 5931 generic.go:356] "Generic (PLEG): container finished" podID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerID="da8587b75d8e48c50d89966c2fdc7257921b3ff5fccb3a68e008c86509a5bd92" exitCode=0 Nov 03 21:41:27.458446 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.458309 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" event={"ID":"09bffbc6-20e7-4e8b-b76f-741a027436c9","Type":"ContainerDied","Data":"da8587b75d8e48c50d89966c2fdc7257921b3ff5fccb3a68e008c86509a5bd92"} Nov 03 21:41:27.515065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.515019 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:41:27.630143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630114 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-podinfo\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630170 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-socket\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630221 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bffbc6-20e7-4e8b-b76f-741a027436c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bffbc6-20e7-4e8b-b76f-741a027436c9-istiod-ca-cert\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630253 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-certs\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630336 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-kube-api-access-slj5m\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-kube-api-access-slj5m\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630391 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-data\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630426 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-credential-socket\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630479 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-envoy\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630508 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-socket" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:27.630698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630524 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-token\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-token\") pod \"09bffbc6-20e7-4e8b-b76f-741a027436c9\" (UID: \"09bffbc6-20e7-4e8b-b76f-741a027436c9\") " Nov 03 21:41:27.630698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630652 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-certs" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:27.630908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630743 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bffbc6-20e7-4e8b-b76f-741a027436c9-istiod-ca-cert" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:41:27.630960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630918 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-data" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:27.630960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.630933 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-credential-socket" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:27.631573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.631338 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.631573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.631370 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.631573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.631385 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.631573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.631398 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bffbc6-20e7-4e8b-b76f-741a027436c9-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.631573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.631412 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.632941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.632918 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-token" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:27.633183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.633163 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-podinfo" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:41:27.633309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.633294 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-envoy" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:27.633346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.633309 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-kube-api-access-slj5m" pod "09bffbc6-20e7-4e8b-b76f-741a027436c9" (UID: "09bffbc6-20e7-4e8b-b76f-741a027436c9"). InnerVolumeSpecName "kube-api-access-slj5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:27.732508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.732479 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.732508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.732505 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.732732 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.732515 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bffbc6-20e7-4e8b-b76f-741a027436c9-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.732732 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.732525 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slj5m\" (UniqueName: \"kubernetes.io/projected/09bffbc6-20e7-4e8b-b76f-741a027436c9-kube-api-access-slj5m\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:41:27.871960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:27.871933 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:28.115798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:28.112850 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:28.466534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:28.466502 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" event={"ID":"09bffbc6-20e7-4e8b-b76f-741a027436c9","Type":"ContainerDied","Data":"511243cb50e4ac2de30a81d3d5b5172e011ed6687f18b8a767150a47cb10b5be"} Nov 03 21:41:28.466534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:28.466528 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792" Nov 03 21:41:28.467028 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:28.466562 5931 scope.go:122] "RemoveContainer" containerID="da8587b75d8e48c50d89966c2fdc7257921b3ff5fccb3a68e008c86509a5bd92" Nov 03 21:41:28.631246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:28.631207 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792"] Nov 03 21:41:28.680199 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:28.680175 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-d4792"] Nov 03 21:41:30.149459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:30.149422 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:30.506778 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:30.506703 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" path="/var/lib/kubelet/pods/09bffbc6-20e7-4e8b-b76f-741a027436c9/volumes" Nov 03 21:41:33.069248 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:41:33.069215 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:41:35.425834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:35.425796 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:40.166605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:40.166563 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:44.524205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:44.524170 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:44.842848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:44.842818 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:49.750112 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:49.750071 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:54.409556 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:54.409521 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:59.148667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:41:59.148594 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:03.067747 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:03.067710 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:42:03.980349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:03.980313 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:07.017787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:07.017750 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:08.854350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:08.854313 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:12.579408 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:12.579374 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:13.925328 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:13.925288 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:17.400230 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:17.400194 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s"] Nov 03 21:42:17.401064 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:17.401029 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" containerID="cri-o://8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2" gracePeriod=30 Nov 03 21:42:18.342744 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.342709 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf"] Nov 03 21:42:18.342957 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.342916 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" podUID="cafa7e11-4065-4d67-b27b-92697f0023af" containerName="maas-api" containerID="cri-o://44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5" gracePeriod=30 Nov 03 21:42:18.555602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.555517 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:18.630577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.630551 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:42:18.712513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.712481 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-metrics-certs\") pod \"cafa7e11-4065-4d67-b27b-92697f0023af\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " Nov 03 21:42:18.712718 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.712543 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafa7e11-4065-4d67-b27b-92697f0023af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafa7e11-4065-4d67-b27b-92697f0023af-openshift-service-ca\") pod \"cafa7e11-4065-4d67-b27b-92697f0023af\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " Nov 03 21:42:18.712718 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.712697 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls\") pod \"cafa7e11-4065-4d67-b27b-92697f0023af\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " Nov 03 21:42:18.712853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.712837 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafa7e11-4065-4d67-b27b-92697f0023af-kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/cafa7e11-4065-4d67-b27b-92697f0023af-kube-api-access-f2kkk\") pod \"cafa7e11-4065-4d67-b27b-92697f0023af\" (UID: \"cafa7e11-4065-4d67-b27b-92697f0023af\") " Nov 03 21:42:18.712924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.712902 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafa7e11-4065-4d67-b27b-92697f0023af-openshift-service-ca" pod "cafa7e11-4065-4d67-b27b-92697f0023af" (UID: "cafa7e11-4065-4d67-b27b-92697f0023af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:18.713416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.713347 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafa7e11-4065-4d67-b27b-92697f0023af-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:18.714904 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.714878 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls" pod "cafa7e11-4065-4d67-b27b-92697f0023af" (UID: "cafa7e11-4065-4d67-b27b-92697f0023af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:42:18.715021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.714922 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-metrics-certs" pod "cafa7e11-4065-4d67-b27b-92697f0023af" (UID: "cafa7e11-4065-4d67-b27b-92697f0023af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:42:18.715078 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.715035 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafa7e11-4065-4d67-b27b-92697f0023af-kube-api-access-f2kkk" pod "cafa7e11-4065-4d67-b27b-92697f0023af" (UID: "cafa7e11-4065-4d67-b27b-92697f0023af"). InnerVolumeSpecName "kube-api-access-f2kkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:18.798058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.798026 5931 generic.go:356] "Generic (PLEG): container finished" podID="cafa7e11-4065-4d67-b27b-92697f0023af" containerID="44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5" exitCode=0 Nov 03 21:42:18.798206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.798071 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" event={"ID":"cafa7e11-4065-4d67-b27b-92697f0023af","Type":"ContainerDied","Data":"44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5"} Nov 03 21:42:18.798206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.798118 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" event={"ID":"cafa7e11-4065-4d67-b27b-92697f0023af","Type":"ContainerDied","Data":"0ae0134536443e71c5454be5e15b3536017444f0ae7cebbf34534f991eea564a"} Nov 03 21:42:18.798206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.798073 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf" Nov 03 21:42:18.798206 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.798145 5931 scope.go:122] "RemoveContainer" containerID="44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5" Nov 03 21:42:18.808915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.808897 5931 scope.go:122] "RemoveContainer" containerID="44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5" Nov 03 21:42:18.809184 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:18.809159 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5\": container with ID starting with 44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5 not found: ID does not exist" containerID="44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5" Nov 03 21:42:18.809237 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.809194 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5"} err="failed to get container status \"44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5\": rpc error: code = NotFound desc = could not find container \"44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5\": container with ID starting with 44c58d3c51bda90e287f01bdded11d548abbbb7d891a3d941bafc99f335eb8a5 not found: ID does not exist" Nov 03 21:42:18.815157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.815139 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:18.815233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.815159 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2kkk\" (UniqueName: \"kubernetes.io/projected/cafa7e11-4065-4d67-b27b-92697f0023af-kube-api-access-f2kkk\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:18.815233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.815169 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cafa7e11-4065-4d67-b27b-92697f0023af-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:18.969043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.968946 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf"] Nov 03 21:42:18.972776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:18.972747 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29472d-7bd6c6549d-bglnf"] Nov 03 21:42:19.370388 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.370344 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc"] Nov 03 21:42:19.371756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.371556 5931 cpu_manager.go:414] "containerMap: removing container" podUID="cafa7e11-4065-4d67-b27b-92697f0023af" containerName="maas-api" Nov 03 21:42:19.371931 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.371912 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafa7e11-4065-4d67-b27b-92697f0023af" containerName="maas-api" Nov 03 21:42:19.372053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.372042 5931 cpu_manager.go:414] "containerMap: removing container" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" Nov 03 21:42:19.372127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.372116 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" Nov 03 21:42:19.372389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.372365 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafa7e11-4065-4d67-b27b-92697f0023af" containerName="maas-api" Nov 03 21:42:19.372505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.372394 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bffbc6-20e7-4e8b-b76f-741a027436c9" containerName="istio-proxy" Nov 03 21:42:19.378235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.378209 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.422981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.422943 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423027 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423076 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423130 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqxh\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-kube-api-access-4dqxh\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423173 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423233 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423297 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423447 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423327 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.423447 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.423427 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eedfea8-2d8c-4598-badd-d1063cc88c09-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.432668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.432066 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc"] Nov 03 21:42:19.433851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.433380 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c1662df5-openshift-default-dockercfg-g9tfw\"" Nov 03 21:42:19.524700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.524670 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.524854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.524711 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.524935 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.524873 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqxh\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-kube-api-access-4dqxh\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.524930 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525006 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525043 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525155 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525094 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525155 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525067 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525159 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525254 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eedfea8-2d8c-4598-badd-d1063cc88c09-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525281 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525408 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525374 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525493 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525474 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.525903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.525885 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eedfea8-2d8c-4598-badd-d1063cc88c09-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.527356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.527340 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.527664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.527643 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.583545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.583516 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqxh\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-kube-api-access-4dqxh\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.583848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.583517 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:19.728579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:19.728490 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:20.141624 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.141581 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:20.160596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.160559 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc"] Nov 03 21:42:20.201411 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:42:20.201385 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eedfea8_2d8c_4598_badd_d1063cc88c09.slice/crio-23df68d951f31daec61d9da3d48cbd33c751eaa135d88db8c5801dd0ab2079b6 WatchSource:0}: Error finding container 23df68d951f31daec61d9da3d48cbd33c751eaa135d88db8c5801dd0ab2079b6: Status 404 returned error can't find the container with id 23df68d951f31daec61d9da3d48cbd33c751eaa135d88db8c5801dd0ab2079b6 Nov 03 21:42:20.203768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.203736 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:20.203857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.203797 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:20.203857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.203847 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:20.457904 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.457802 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:20.512633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.511833 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafa7e11-4065-4d67-b27b-92697f0023af" path="/var/lib/kubelet/pods/cafa7e11-4065-4d67-b27b-92697f0023af/volumes" Nov 03 21:42:20.685022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.684990 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:20.856047 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.856013 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" event={"ID":"6eedfea8-2d8c-4598-badd-d1063cc88c09","Type":"ContainerStarted","Data":"105a9c5ecdbc69a95b18261a418101fdfdf0f85706830f1800df687743d673db"} Nov 03 21:42:20.856224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:20.856069 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" event={"ID":"6eedfea8-2d8c-4598-badd-d1063cc88c09","Type":"ContainerStarted","Data":"23df68d951f31daec61d9da3d48cbd33c751eaa135d88db8c5801dd0ab2079b6"} Nov 03 21:42:21.020487 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:21.020432 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" podStartSLOduration=2.020413297 podStartE2EDuration="2.020413297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:21.019880941 +0000 UTC m=+1608.630728755" watchObservedRunningTime="2026-11-03 21:42:21.020413297 +0000 UTC m=+1608.631261112" Nov 03 21:42:21.247427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:21.247322 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:21.739634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:21.734879 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:21.746734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:21.746705 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:42:21.746850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:21.746764 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 21:42:22.410667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.410638 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:42:22.410848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.410687 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:42:22.741636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.740996 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:42:22.747641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.741065 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 21:42:22.747923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.747902 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:42:22.775068 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775048 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-socket\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775131 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-data\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775163 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-istio-token\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-istio-token\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775207 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-credential-socket\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775232 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-kube-api-access-8zrcb\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775255 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-envoy\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775272 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-certs\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775292 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-socket" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:22.775329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775297 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b22394b-c087-43f1-8a37-862938a4830b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b22394b-c087-43f1-8a37-862938a4830b-istio-podinfo\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775399 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b22394b-c087-43f1-8a37-862938a4830b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b22394b-c087-43f1-8a37-862938a4830b-istiod-ca-cert\") pod \"0b22394b-c087-43f1-8a37-862938a4830b\" (UID: \"0b22394b-c087-43f1-8a37-862938a4830b\") " Nov 03 21:42:22.775584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775427 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-data" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:22.775738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775662 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-certs" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:22.775738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775683 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-credential-socket" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:22.775890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.775860 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b22394b-c087-43f1-8a37-862938a4830b-istiod-ca-cert" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:22.776287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.776237 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.776442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.776419 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.776539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.776447 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.776539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.776463 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.776539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.776478 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b22394b-c087-43f1-8a37-862938a4830b-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.777846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.777824 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-istio-token" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:22.777973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.777827 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b22394b-c087-43f1-8a37-862938a4830b-istio-podinfo" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:42:22.777973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.777882 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-envoy" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:22.777973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.777893 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-kube-api-access-8zrcb" pod "0b22394b-c087-43f1-8a37-862938a4830b" (UID: "0b22394b-c087-43f1-8a37-862938a4830b"). InnerVolumeSpecName "kube-api-access-8zrcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:22.870126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.870091 5931 generic.go:356] "Generic (PLEG): container finished" podID="0b22394b-c087-43f1-8a37-862938a4830b" containerID="8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2" exitCode=0 Nov 03 21:42:22.870285 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.870133 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" Nov 03 21:42:22.870285 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.870181 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" event={"ID":"0b22394b-c087-43f1-8a37-862938a4830b","Type":"ContainerDied","Data":"8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2"} Nov 03 21:42:22.870285 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.870233 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s" event={"ID":"0b22394b-c087-43f1-8a37-862938a4830b","Type":"ContainerDied","Data":"e55fc69834f5213f20ee4164ad51be9a97e05785b251de652ca8a6cf2af6db13"} Nov 03 21:42:22.870285 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.870260 5931 scope.go:122] "RemoveContainer" containerID="8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2" Nov 03 21:42:22.878118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.878097 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.878202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.878121 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/0b22394b-c087-43f1-8a37-862938a4830b-kube-api-access-8zrcb\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.878202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.878131 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b22394b-c087-43f1-8a37-862938a4830b-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.878202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.878140 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b22394b-c087-43f1-8a37-862938a4830b-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:22.881104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.881086 5931 scope.go:122] "RemoveContainer" containerID="8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2" Nov 03 21:42:22.881347 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:22.881327 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2\": container with ID starting with 8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2 not found: ID does not exist" containerID="8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2" Nov 03 21:42:22.881391 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:22.881355 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2"} err="failed to get container status \"8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2\": rpc error: code = NotFound desc = could not find container \"8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2\": container with ID starting with 8844901b21bf2d3319bda0003bfc158e40f7c31ce186e50031f87a5d92f5fcd2 not found: ID does not exist" Nov 03 21:42:23.051032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:23.050953 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s"] Nov 03 21:42:23.054094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:23.054073 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-cv29s"] Nov 03 21:42:23.736645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:23.735918 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:42:23.736645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:23.735975 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 21:42:24.506987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:24.506956 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b22394b-c087-43f1-8a37-862938a4830b" path="/var/lib/kubelet/pods/0b22394b-c087-43f1-8a37-862938a4830b/volumes" Nov 03 21:42:24.736634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:24.735970 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:42:24.736634 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:24.736029 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 21:42:25.473999 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:25.473960 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:25.699420 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:25.699392 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:25.745682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:25.741936 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:42:25.745682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:25.741993 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 21:42:25.936414 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:25.936371 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:26.844647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:26.844604 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:26.845120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:26.845101 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:26.846274 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:26.846254 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:42:27.142907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:27.142829 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:27.380229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:27.380203 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:27.601266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:27.601230 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:30.911712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:30.911672 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:31.841430 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:31.841399 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp"] Nov 03 21:42:32.194164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:32.194062 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:32.430237 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:32.430208 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:33.068942 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:33.068901 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:42:34.379725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.379695 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf"] Nov 03 21:42:34.380332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.380320 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" Nov 03 21:42:34.380378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.380334 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" Nov 03 21:42:34.380445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.380437 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b22394b-c087-43f1-8a37-862938a4830b" containerName="istio-proxy" Nov 03 21:42:34.385266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.385237 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.429480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.429454 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95"] Nov 03 21:42:34.433448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.433433 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.468240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.468215 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf"] Nov 03 21:42:34.528230 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.528203 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95"] Nov 03 21:42:34.543159 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543134 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543173 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543375 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543331 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543382 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543420 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543467 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543490 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543508 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543648 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543677 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543710 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543737 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543841 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thf8z\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-kube-api-access-thf8z\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543865 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543916 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.543998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543942 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.543998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.543978 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.544100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.544074 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g527\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-kube-api-access-2g527\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.644971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.644901 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g527\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-kube-api-access-2g527\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.644971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.644948 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.644971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.644971 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.645187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645023 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645046 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.645187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645063 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.645187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645120 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645143 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645159 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645207 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645222 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645257 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645315 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645381 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thf8z\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-kube-api-access-thf8z\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645399 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645424 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645449 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.645854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645505 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645515 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645822 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.645854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645828 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.646056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.645986 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.646113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.646050 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.646324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.646282 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.646324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.646289 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.646324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.646284 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.646548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.646523 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.646709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.646686 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.647965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.647941 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.648070 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.648042 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.648124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.648099 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.648384 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.648370 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.700998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.700979 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thf8z\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-kube-api-access-thf8z\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.701169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.701150 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g527\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-kube-api-access-2g527\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.704224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.704204 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:34.704357 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.704338 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.746150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.746129 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:34.999163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:34.999079 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:35.178182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.178158 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95"] Nov 03 21:42:35.179876 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:42:35.179851 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9931e2f_5857_4964_b99b_fe8eb3b8a281.slice/crio-c0851356173772d148a49da0fff624c095c67bf3591e3987064e01dd9730100a WatchSource:0}: Error finding container c0851356173772d148a49da0fff624c095c67bf3591e3987064e01dd9730100a: Status 404 returned error can't find the container with id c0851356173772d148a49da0fff624c095c67bf3591e3987064e01dd9730100a Nov 03 21:42:35.182712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.182673 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:35.182775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.182741 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:35.182815 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.182775 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:35.279832 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.279803 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc"] Nov 03 21:42:35.280085 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.280035 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" podUID="5f243fff-cdba-45cd-8096-397653259359" containerName="maas-api" containerID="cri-o://44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98" gracePeriod=30 Nov 03 21:42:35.378733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.378707 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf"] Nov 03 21:42:35.381377 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:42:35.381346 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8efb056_b5a5_4098_b8ea_dac1bc7f1522.slice/crio-d7c02856e2740f1c760a7398a6601d21f453c615e46da3aa621da97477bd83f7 WatchSource:0}: Error finding container d7c02856e2740f1c760a7398a6601d21f453c615e46da3aa621da97477bd83f7: Status 404 returned error can't find the container with id d7c02856e2740f1c760a7398a6601d21f453c615e46da3aa621da97477bd83f7 Nov 03 21:42:35.386517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.386464 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:35.386762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.386639 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:35.386762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.386729 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:42:35.515521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.507073 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:35.570041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.569933 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:42:35.658515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.658473 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f243fff-cdba-45cd-8096-397653259359-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f243fff-cdba-45cd-8096-397653259359-openshift-service-ca\") pod \"5f243fff-cdba-45cd-8096-397653259359\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " Nov 03 21:42:35.658699 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.658584 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f243fff-cdba-45cd-8096-397653259359-kube-api-access-v4ddf\" (UniqueName: \"kubernetes.io/projected/5f243fff-cdba-45cd-8096-397653259359-kube-api-access-v4ddf\") pod \"5f243fff-cdba-45cd-8096-397653259359\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " Nov 03 21:42:35.658699 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.658673 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls\") pod \"5f243fff-cdba-45cd-8096-397653259359\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " Nov 03 21:42:35.658823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.658798 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-metrics-certs\") pod \"5f243fff-cdba-45cd-8096-397653259359\" (UID: \"5f243fff-cdba-45cd-8096-397653259359\") " Nov 03 21:42:35.658907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.658883 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f243fff-cdba-45cd-8096-397653259359-openshift-service-ca" pod "5f243fff-cdba-45cd-8096-397653259359" (UID: "5f243fff-cdba-45cd-8096-397653259359"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:35.659459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.659436 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f243fff-cdba-45cd-8096-397653259359-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:35.661415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.661374 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f243fff-cdba-45cd-8096-397653259359-kube-api-access-v4ddf" pod "5f243fff-cdba-45cd-8096-397653259359" (UID: "5f243fff-cdba-45cd-8096-397653259359"). InnerVolumeSpecName "kube-api-access-v4ddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:35.662861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.661988 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-metrics-certs" pod "5f243fff-cdba-45cd-8096-397653259359" (UID: "5f243fff-cdba-45cd-8096-397653259359"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:42:35.663038 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.663017 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls" pod "5f243fff-cdba-45cd-8096-397653259359" (UID: "5f243fff-cdba-45cd-8096-397653259359"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:42:35.761580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.761550 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4ddf\" (UniqueName: \"kubernetes.io/projected/5f243fff-cdba-45cd-8096-397653259359-kube-api-access-v4ddf\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:35.761580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.761579 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:35.761812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.761588 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f243fff-cdba-45cd-8096-397653259359-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:35.962541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.962503 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" event={"ID":"e8efb056-b5a5-4098-b8ea-dac1bc7f1522","Type":"ContainerStarted","Data":"1132d54da47b7fe44c8e4ae442a4401fc9e4226e11f6ab74c68281de13956684"} Nov 03 21:42:35.962734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.962604 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" event={"ID":"e8efb056-b5a5-4098-b8ea-dac1bc7f1522","Type":"ContainerStarted","Data":"d7c02856e2740f1c760a7398a6601d21f453c615e46da3aa621da97477bd83f7"} Nov 03 21:42:35.964301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.964273 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" event={"ID":"b9931e2f-5857-4964-b99b-fe8eb3b8a281","Type":"ContainerStarted","Data":"3526c912c77fdf6c2f041c19530199818951d55850aac26d61e84f3122db4557"} Nov 03 21:42:35.964400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.964328 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" event={"ID":"b9931e2f-5857-4964-b99b-fe8eb3b8a281","Type":"ContainerStarted","Data":"c0851356173772d148a49da0fff624c095c67bf3591e3987064e01dd9730100a"} Nov 03 21:42:35.965855 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.965834 5931 generic.go:356] "Generic (PLEG): container finished" podID="5f243fff-cdba-45cd-8096-397653259359" containerID="44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98" exitCode=0 Nov 03 21:42:35.965953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.965868 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" Nov 03 21:42:35.965953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.965889 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" event={"ID":"5f243fff-cdba-45cd-8096-397653259359","Type":"ContainerDied","Data":"44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98"} Nov 03 21:42:35.965953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.965921 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc" event={"ID":"5f243fff-cdba-45cd-8096-397653259359","Type":"ContainerDied","Data":"26e96a4fc8a3cb74a3fa9f12c492b240c223ade89af576d932fb046a1f177f3e"} Nov 03 21:42:35.965953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.965943 5931 scope.go:122] "RemoveContainer" containerID="44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98" Nov 03 21:42:35.979543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.979523 5931 scope.go:122] "RemoveContainer" containerID="44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98" Nov 03 21:42:35.979873 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:35.979848 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98\": container with ID starting with 44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98 not found: ID does not exist" containerID="44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98" Nov 03 21:42:35.979959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:35.979886 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98"} err="failed to get container status \"44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98\": rpc error: code = NotFound desc = could not find container \"44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98\": container with ID starting with 44d0a78b341d6cbc6b0acd5090b1976a0ee40912aae1b022ce68066eb5bd6b98 not found: ID does not exist" Nov 03 21:42:36.011724 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.009784 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:36.012060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.012030 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:42:36.012261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.012234 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:42:36.125087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.125023 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podStartSLOduration=2.125004067 podStartE2EDuration="2.125004067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:36.089064947 +0000 UTC m=+1623.699912762" watchObservedRunningTime="2026-11-03 21:42:36.125004067 +0000 UTC m=+1623.735851882" Nov 03 21:42:36.206518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.206467 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podStartSLOduration=2.206364371 podStartE2EDuration="2.206364371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:36.204126049 +0000 UTC m=+1623.814973866" watchObservedRunningTime="2026-11-03 21:42:36.206364371 +0000 UTC m=+1623.817212186" Nov 03 21:42:36.319075 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.318971 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc"] Nov 03 21:42:36.340267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.340232 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5025c-6d88ddf7c8-dv4nc"] Nov 03 21:42:36.509355 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.509321 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f243fff-cdba-45cd-8096-397653259359" path="/var/lib/kubelet/pods/5f243fff-cdba-45cd-8096-397653259359/volumes" Nov 03 21:42:36.752749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.752662 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:36.773293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.773240 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:42:36.773507 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.773316 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:42:36.845637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:36.844193 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" podUID="cddceccb-7d62-43b5-abad-da056ea7fc0c" containerName="payload-pre-processing" containerID="cri-o://95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f" gracePeriod=25 Nov 03 21:42:37.002484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.002459 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:42:37.002641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.002503 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:42:37.181635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.180112 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:37.274783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.274403 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:42:37.293386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.293351 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cddceccb-7d62-43b5-abad-da056ea7fc0c-kube-api-access-7896m\" (UniqueName: \"kubernetes.io/projected/cddceccb-7d62-43b5-abad-da056ea7fc0c-kube-api-access-7896m\") pod \"cddceccb-7d62-43b5-abad-da056ea7fc0c\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " Nov 03 21:42:37.293540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.293460 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cddceccb-7d62-43b5-abad-da056ea7fc0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cddceccb-7d62-43b5-abad-da056ea7fc0c-plugins-config-volume\") pod \"cddceccb-7d62-43b5-abad-da056ea7fc0c\" (UID: \"cddceccb-7d62-43b5-abad-da056ea7fc0c\") " Nov 03 21:42:37.293997 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.293973 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddceccb-7d62-43b5-abad-da056ea7fc0c-plugins-config-volume" pod "cddceccb-7d62-43b5-abad-da056ea7fc0c" (UID: "cddceccb-7d62-43b5-abad-da056ea7fc0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:37.295537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.295517 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddceccb-7d62-43b5-abad-da056ea7fc0c-kube-api-access-7896m" pod "cddceccb-7d62-43b5-abad-da056ea7fc0c" (UID: "cddceccb-7d62-43b5-abad-da056ea7fc0c"). InnerVolumeSpecName "kube-api-access-7896m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:37.395487 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.395460 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7896m\" (UniqueName: \"kubernetes.io/projected/cddceccb-7d62-43b5-abad-da056ea7fc0c-kube-api-access-7896m\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:37.395695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.395673 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cddceccb-7d62-43b5-abad-da056ea7fc0c-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:37.748688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.748073 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:42:37.748688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.748131 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:42:37.831431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.831399 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:37.995939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.995902 5931 generic.go:356] "Generic (PLEG): container finished" podID="cddceccb-7d62-43b5-abad-da056ea7fc0c" containerID="95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f" exitCode=0 Nov 03 21:42:37.996110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.995984 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" event={"ID":"cddceccb-7d62-43b5-abad-da056ea7fc0c","Type":"ContainerDied","Data":"95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f"} Nov 03 21:42:37.996110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.996049 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" event={"ID":"cddceccb-7d62-43b5-abad-da056ea7fc0c","Type":"ContainerDied","Data":"9dc603b02d0659077182dc0171f89e7b8ca72a1f4186eddc29535e9dfcf22ea4"} Nov 03 21:42:37.996110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.996078 5931 scope.go:122] "RemoveContainer" containerID="95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f" Nov 03 21:42:37.996378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.996361 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp" Nov 03 21:42:37.999527 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.999468 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:42:37.999636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:37.999530 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:42:38.007115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.007084 5931 scope.go:122] "RemoveContainer" containerID="95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f" Nov 03 21:42:38.007394 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:38.007376 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f\": container with ID starting with 95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f not found: ID does not exist" containerID="95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f" Nov 03 21:42:38.007478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.007405 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f"} err="failed to get container status \"95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f\": rpc error: code = NotFound desc = could not find container \"95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f\": container with ID starting with 95ff5deaf6611e4593bbf9220f9324e73a43275ee04732e7e3c63f8926d6557f not found: ID does not exist" Nov 03 21:42:38.095086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.095058 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:38.177519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.177480 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp"] Nov 03 21:42:38.182107 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.182088 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30f143-698f694d7f-wblqp"] Nov 03 21:42:38.325701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.323796 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:38.513432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.513397 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cddceccb-7d62-43b5-abad-da056ea7fc0c" path="/var/lib/kubelet/pods/cddceccb-7d62-43b5-abad-da056ea7fc0c/volumes" Nov 03 21:42:38.764637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.749006 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:42:38.764637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.749064 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:42:38.999917 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.999885 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:42:39.000097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:38.999944 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:42:39.747003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:39.746966 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:42:39.747250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:39.747224 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:42:40.000176 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:40.000097 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:42:40.000176 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:40.000153 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:42:40.299824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:40.299722 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:40.545643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:40.545592 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:40.751906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:40.751865 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:42:40.752150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:40.752120 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:42:41.001849 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:41.001810 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:42:41.002360 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:41.001867 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:42:41.752106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:41.752069 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:42:41.752397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:41.752367 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:42:41.873786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:41.873755 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:41.999548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:41.999516 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:42:41.999763 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:41.999563 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 21:42:42.138266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:42.137509 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:42.750748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:42.750721 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:42.751122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:42.751106 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:42.751858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:42.751834 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:42:43.003942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:43.003879 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:43.004158 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:43.004139 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:43.004785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:43.004770 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:42:43.092739 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:43.092685 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:43.954801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:43.954765 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:44.199990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:44.199962 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt"] Nov 03 21:42:44.200227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:44.200191 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" containerID="cri-o://49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3" gracePeriod=30 Nov 03 21:42:44.226658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:44.226582 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:44.454099 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:44.454069 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:44.895531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:44.895502 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:45.679659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:45.679629 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd"] Nov 03 21:42:45.680018 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:45.679798 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" containerID="cri-o://1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a" gracePeriod=30 Nov 03 21:42:45.729506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:45.729465 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5"] Nov 03 21:42:45.729801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:45.729762 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" containerID="cri-o://5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567" gracePeriod=30 Nov 03 21:42:45.732946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:45.732922 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv"] Nov 03 21:42:45.733157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:45.733128 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" containerID="cri-o://f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb" gracePeriod=30 Nov 03 21:42:47.361703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:47.361671 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn"] Nov 03 21:42:47.472994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:47.472963 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l"] Nov 03 21:42:47.689848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:47.689761 5931 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:42:47.690307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:47.690263 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:42:47.848071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:47.846888 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:48.082719 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:48.082645 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:48.385062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:48.385026 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:49.540368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.540346 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:42:49.685496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685468 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-credential-socket\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.685662 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685511 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istiod-ca-cert\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.685662 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685554 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-envoy\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.685786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685756 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-credential-socket" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:49.685836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685793 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-certs\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.685884 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685874 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-token\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-token\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.685939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685902 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-data\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.685939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685929 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istiod-ca-cert" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:49.686032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.685983 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-socket\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.686032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.686022 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-kube-api-access-2lqxp\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-kube-api-access-2lqxp\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.686117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.686024 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-certs" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:49.686171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.686116 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-podinfo\") pod \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\" (UID: \"6d4287f3-df53-47db-874c-3e8d3bfd92f6\") " Nov 03 21:42:49.686226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.686180 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-socket" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:49.686226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.686194 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-data" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:49.687410 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.687386 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.687535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.687415 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.687535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.687432 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.687535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.687445 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.687535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.687457 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.688152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.688128 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-envoy" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:49.688262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.688245 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-token" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:49.688370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.688351 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-kube-api-access-2lqxp" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "kube-api-access-2lqxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:49.688526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.688511 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-podinfo" pod "6d4287f3-df53-47db-874c-3e8d3bfd92f6" (UID: "6d4287f3-df53-47db-874c-3e8d3bfd92f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:42:49.789397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.789334 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.789397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.789358 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.789397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.789368 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:49.789397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:49.789376 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lqxp\" (UniqueName: \"kubernetes.io/projected/6d4287f3-df53-47db-874c-3e8d3bfd92f6-kube-api-access-2lqxp\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:50.115340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.115313 5931 generic.go:356] "Generic (PLEG): container finished" podID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerID="49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3" exitCode=0 Nov 03 21:42:50.115470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.115411 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" Nov 03 21:42:50.115470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.115424 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" event={"ID":"6d4287f3-df53-47db-874c-3e8d3bfd92f6","Type":"ContainerDied","Data":"49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3"} Nov 03 21:42:50.115535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.115493 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt" event={"ID":"6d4287f3-df53-47db-874c-3e8d3bfd92f6","Type":"ContainerDied","Data":"59a020874888d0e0d051eae778958a6647560c79b77bd9a4bdebfa2669759bc5"} Nov 03 21:42:50.115535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.115523 5931 scope.go:122] "RemoveContainer" containerID="49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3" Nov 03 21:42:50.126478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.126463 5931 scope.go:122] "RemoveContainer" containerID="8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271" Nov 03 21:42:50.136156 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.136137 5931 scope.go:122] "RemoveContainer" containerID="49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3" Nov 03 21:42:50.136403 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:50.136382 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3\": container with ID starting with 49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3 not found: ID does not exist" containerID="49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3" Nov 03 21:42:50.136480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.136414 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3"} err="failed to get container status \"49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3\": rpc error: code = NotFound desc = could not find container \"49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3\": container with ID starting with 49f1e675ffa9459fefad49a2ea4c5114d350e18642c204b00306fe49ec705ed3 not found: ID does not exist" Nov 03 21:42:50.136480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.136437 5931 scope.go:122] "RemoveContainer" containerID="8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271" Nov 03 21:42:50.136667 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:50.136651 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271\": container with ID starting with 8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271 not found: ID does not exist" containerID="8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271" Nov 03 21:42:50.136728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.136674 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271"} err="failed to get container status \"8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271\": rpc error: code = NotFound desc = could not find container \"8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271\": container with ID starting with 8eea69c8b81d80acf5dde16829446e8b7b349ef1e5cca359481402acfbc6c271 not found: ID does not exist" Nov 03 21:42:50.280806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.280781 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt"] Nov 03 21:42:50.286252 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.286231 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c5025c-openshift-default-8964458d9-w25qt"] Nov 03 21:42:50.506711 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:50.506644 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" path="/var/lib/kubelet/pods/6d4287f3-df53-47db-874c-3e8d3bfd92f6/volumes" Nov 03 21:42:51.018663 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.018601 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:42:51.041513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.041491 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:42:51.072011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.071992 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:42:51.124142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.124119 5931 generic.go:356] "Generic (PLEG): container finished" podID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerID="1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a" exitCode=0 Nov 03 21:42:51.124259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.124167 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" Nov 03 21:42:51.124259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.124202 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" event={"ID":"c141d35b-42f3-42c7-9877-4e918a21b4dc","Type":"ContainerDied","Data":"1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a"} Nov 03 21:42:51.124259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.124249 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd" event={"ID":"c141d35b-42f3-42c7-9877-4e918a21b4dc","Type":"ContainerDied","Data":"636ead21949add8cb6d80a4092b86ff5633fbd1c27b7097b1a4823a305ac1f4a"} Nov 03 21:42:51.124405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.124275 5931 scope.go:122] "RemoveContainer" containerID="1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a" Nov 03 21:42:51.125656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.125605 5931 generic.go:356] "Generic (PLEG): container finished" podID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerID="f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb" exitCode=0 Nov 03 21:42:51.125656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.125655 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" Nov 03 21:42:51.125975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.125721 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" event={"ID":"0c895147-e359-4fce-ab8a-f58a70f58b82","Type":"ContainerDied","Data":"f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb"} Nov 03 21:42:51.125975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.125759 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv" event={"ID":"0c895147-e359-4fce-ab8a-f58a70f58b82","Type":"ContainerDied","Data":"3c829ce08f4ab85ffa51d2397845f995c285c0b8b77d89bd6f99775f223f42d2"} Nov 03 21:42:51.127139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.127121 5931 generic.go:356] "Generic (PLEG): container finished" podID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerID="5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567" exitCode=0 Nov 03 21:42:51.127249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.127159 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" Nov 03 21:42:51.127249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.127213 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" event={"ID":"de70907b-7e1b-462e-beb9-50b66b931bf9","Type":"ContainerDied","Data":"5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567"} Nov 03 21:42:51.127358 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.127252 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5" event={"ID":"de70907b-7e1b-462e-beb9-50b66b931bf9","Type":"ContainerDied","Data":"e90131fc6df54ebf4e87551b4ee7d87a6a7a3e1131aac0bce37730f30ba681bf"} Nov 03 21:42:51.135562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.135544 5931 scope.go:122] "RemoveContainer" containerID="1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a" Nov 03 21:42:51.135923 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:51.135887 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a\": container with ID starting with 1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a not found: ID does not exist" containerID="1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a" Nov 03 21:42:51.136016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.135922 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a"} err="failed to get container status \"1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a\": rpc error: code = NotFound desc = could not find container \"1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a\": container with ID starting with 1a0bb1a1f12d557879b75f32d87de60292c5d9802c848ba6f9df4645b86a515a not found: ID does not exist" Nov 03 21:42:51.136016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.135945 5931 scope.go:122] "RemoveContainer" containerID="f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb" Nov 03 21:42:51.149627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.149590 5931 scope.go:122] "RemoveContainer" containerID="f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb" Nov 03 21:42:51.149972 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:51.149944 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb\": container with ID starting with f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb not found: ID does not exist" containerID="f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb" Nov 03 21:42:51.150083 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.149979 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb"} err="failed to get container status \"f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb\": rpc error: code = NotFound desc = could not find container \"f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb\": container with ID starting with f041057e48f5f2e9db337df486498bafbad9620d677ecf7703f53314b1954fcb not found: ID does not exist" Nov 03 21:42:51.150083 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.149995 5931 scope.go:122] "RemoveContainer" containerID="5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567" Nov 03 21:42:51.171413 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.171390 5931 scope.go:122] "RemoveContainer" containerID="5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567" Nov 03 21:42:51.171757 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:51.171727 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567\": container with ID starting with 5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567 not found: ID does not exist" containerID="5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567" Nov 03 21:42:51.171847 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.171753 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567"} err="failed to get container status \"5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567\": rpc error: code = NotFound desc = could not find container \"5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567\": container with ID starting with 5193424752b4586af4b1368a7ddd08c88b18f63516fb2e36f79645e934e37567 not found: ID does not exist" Nov 03 21:42:51.210399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210376 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-token\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-token\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.210492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210428 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-token\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-token\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.210492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210460 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-certs\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.210648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210493 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-podinfo\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.210648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210536 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-data\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.210648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210567 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-socket\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.210648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210604 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-envoy\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.210853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210670 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-certs\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.210853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210703 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-podinfo\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.210853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210734 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-kube-api-access-mwkvx\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-kube-api-access-mwkvx\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.210853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210764 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-kube-api-access-zsrgf\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-kube-api-access-zsrgf\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.210853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210795 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-socket\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.210853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210784 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-certs" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210862 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-data" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210907 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-data\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.210971 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-data\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211013 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-certs\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211044 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c141d35b-42f3-42c7-9877-4e918a21b4dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c141d35b-42f3-42c7-9877-4e918a21b4dc-istiod-ca-cert\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211105 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-podinfo\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211141 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-socket" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211145 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-token\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-token\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.211331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211218 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de70907b-7e1b-462e-beb9-50b66b931bf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de70907b-7e1b-462e-beb9-50b66b931bf9-istiod-ca-cert\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.211959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211489 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-data" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.211959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211508 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-envoy\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.211959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211560 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-kube-api-access-nzglm\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-kube-api-access-nzglm\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.211959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211680 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-envoy\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.211959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211750 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-credential-socket\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.211959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211790 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c895147-e359-4fce-ab8a-f58a70f58b82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c895147-e359-4fce-ab8a-f58a70f58b82-istiod-ca-cert\") pod \"0c895147-e359-4fce-ab8a-f58a70f58b82\" (UID: \"0c895147-e359-4fce-ab8a-f58a70f58b82\") " Nov 03 21:42:51.211959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211914 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-credential-socket\") pod \"c141d35b-42f3-42c7-9877-4e918a21b4dc\" (UID: \"c141d35b-42f3-42c7-9877-4e918a21b4dc\") " Nov 03 21:42:51.212296 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.211988 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-credential-socket\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.212296 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212063 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-socket\") pod \"de70907b-7e1b-462e-beb9-50b66b931bf9\" (UID: \"de70907b-7e1b-462e-beb9-50b66b931bf9\") " Nov 03 21:42:51.212296 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212119 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-certs" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.212457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212296 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-socket" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.212514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212490 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-socket" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.212589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212557 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c895147-e359-4fce-ab8a-f58a70f58b82-istiod-ca-cert" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:51.212706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212688 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-credential-socket" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.212769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212704 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-certs" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.212841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212820 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-credential-socket" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.212921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212854 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-data" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.213026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.212976 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213021 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213037 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213051 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213066 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213071 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-credential-socket" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213080 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213097 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213111 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213124 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c895147-e359-4fce-ab8a-f58a70f58b82-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213136 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213136 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c141d35b-42f3-42c7-9877-4e918a21b4dc-istiod-ca-cert" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:51.213841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213149 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213164 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.213841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.213654 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de70907b-7e1b-462e-beb9-50b66b931bf9-istiod-ca-cert" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:51.214537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.214499 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-token" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:51.214785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.214762 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-token" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:51.215120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.215077 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-token" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:51.215120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.215102 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-kube-api-access-zsrgf" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "kube-api-access-zsrgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:51.215283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.215119 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-podinfo" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:42:51.215283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.215168 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-podinfo" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:42:51.215530 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.215509 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-kube-api-access-nzglm" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "kube-api-access-nzglm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:51.215585 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.215576 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-envoy" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.215662 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.215604 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-kube-api-access-mwkvx" pod "0c895147-e359-4fce-ab8a-f58a70f58b82" (UID: "0c895147-e359-4fce-ab8a-f58a70f58b82"). InnerVolumeSpecName "kube-api-access-mwkvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:51.216049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.216031 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-envoy" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.216591 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.216567 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-podinfo" pod "de70907b-7e1b-462e-beb9-50b66b931bf9" (UID: "de70907b-7e1b-462e-beb9-50b66b931bf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:42:51.216785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.216772 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-envoy" pod "c141d35b-42f3-42c7-9877-4e918a21b4dc" (UID: "c141d35b-42f3-42c7-9877-4e918a21b4dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:42:51.314154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314083 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314107 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314116 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwkvx\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-kube-api-access-mwkvx\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314125 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrgf\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-kube-api-access-zsrgf\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314134 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c141d35b-42f3-42c7-9877-4e918a21b4dc-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314145 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314160 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314171 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de70907b-7e1b-462e-beb9-50b66b931bf9-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314183 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314196 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzglm\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-kube-api-access-nzglm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314208 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314218 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de70907b-7e1b-462e-beb9-50b66b931bf9-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314227 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de70907b-7e1b-462e-beb9-50b66b931bf9-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314236 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c141d35b-42f3-42c7-9877-4e918a21b4dc-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.314428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.314244 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c895147-e359-4fce-ab8a-f58a70f58b82-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:51.549735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.549702 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd"] Nov 03 21:42:51.553051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.553020 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hkxpd"] Nov 03 21:42:51.660962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.660925 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv"] Nov 03 21:42:51.664833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.664810 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-zlxxv"] Nov 03 21:42:51.772987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.772959 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5"] Nov 03 21:42:51.776833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:51.776813 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-hzvx5"] Nov 03 21:42:52.362180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.362146 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" podUID="c9c8c7fd-bc3a-42d3-807c-f77238e0e732" containerName="payload-processing" containerID="cri-o://7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78" gracePeriod=25 Nov 03 21:42:52.474196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.474152 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" podUID="92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" containerName="payload-pre-processing" containerID="cri-o://d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce" gracePeriod=25 Nov 03 21:42:52.508710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.508683 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" path="/var/lib/kubelet/pods/0c895147-e359-4fce-ab8a-f58a70f58b82/volumes" Nov 03 21:42:52.509160 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.509147 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" path="/var/lib/kubelet/pods/c141d35b-42f3-42c7-9877-4e918a21b4dc/volumes" Nov 03 21:42:52.509541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.509529 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" path="/var/lib/kubelet/pods/de70907b-7e1b-462e-beb9-50b66b931bf9/volumes" Nov 03 21:42:52.636323 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.636301 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:42:52.727874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.727749 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-plugins-config-volume\") pod \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " Nov 03 21:42:52.727874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.727833 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-kube-api-access-wdhmn\" (UniqueName: \"kubernetes.io/projected/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-kube-api-access-wdhmn\") pod \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\" (UID: \"c9c8c7fd-bc3a-42d3-807c-f77238e0e732\") " Nov 03 21:42:52.728756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.728729 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-plugins-config-volume" pod "c9c8c7fd-bc3a-42d3-807c-f77238e0e732" (UID: "c9c8c7fd-bc3a-42d3-807c-f77238e0e732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:52.730640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.730587 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-kube-api-access-wdhmn" pod "c9c8c7fd-bc3a-42d3-807c-f77238e0e732" (UID: "c9c8c7fd-bc3a-42d3-807c-f77238e0e732"). InnerVolumeSpecName "kube-api-access-wdhmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:52.758210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.758185 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:42:52.829186 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.829156 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-kube-api-access-hdgdf\" (UniqueName: \"kubernetes.io/projected/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-kube-api-access-hdgdf\") pod \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " Nov 03 21:42:52.829330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.829212 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-plugins-config-volume\") pod \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\" (UID: \"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1\") " Nov 03 21:42:52.829672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.829648 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-plugins-config-volume" pod "92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" (UID: "92614c7d-1f19-4a6e-95cb-5ba987bd3bb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:42:52.829887 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.829873 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:52.829960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.829890 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:52.829960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.829903 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhmn\" (UniqueName: \"kubernetes.io/projected/c9c8c7fd-bc3a-42d3-807c-f77238e0e732-kube-api-access-wdhmn\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:52.831426 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.831399 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-kube-api-access-hdgdf" pod "92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" (UID: "92614c7d-1f19-4a6e-95cb-5ba987bd3bb1"). InnerVolumeSpecName "kube-api-access-hdgdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:52.931109 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:52.931032 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgdf\" (UniqueName: \"kubernetes.io/projected/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1-kube-api-access-hdgdf\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:42:53.146028 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.145993 5931 generic.go:356] "Generic (PLEG): container finished" podID="c9c8c7fd-bc3a-42d3-807c-f77238e0e732" containerID="7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78" exitCode=0 Nov 03 21:42:53.146204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.146043 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" Nov 03 21:42:53.146204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.146157 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" event={"ID":"c9c8c7fd-bc3a-42d3-807c-f77238e0e732","Type":"ContainerDied","Data":"7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78"} Nov 03 21:42:53.146204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.146200 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn" event={"ID":"c9c8c7fd-bc3a-42d3-807c-f77238e0e732","Type":"ContainerDied","Data":"9196affe963beb055f7e029ea09c17638bd7e5f3b9cb82110df26d844016f647"} Nov 03 21:42:53.146328 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.146224 5931 scope.go:122] "RemoveContainer" containerID="7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78" Nov 03 21:42:53.148410 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.148383 5931 generic.go:356] "Generic (PLEG): container finished" podID="92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" containerID="d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce" exitCode=0 Nov 03 21:42:53.148552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.148487 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" event={"ID":"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1","Type":"ContainerDied","Data":"d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce"} Nov 03 21:42:53.148552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.148531 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" event={"ID":"92614c7d-1f19-4a6e-95cb-5ba987bd3bb1","Type":"ContainerDied","Data":"27c8fe00a5dbbe14cd58463d828e02b347924b18f2a0b5b911a58c554b9f0219"} Nov 03 21:42:53.148798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.148771 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l" Nov 03 21:42:53.160831 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.159970 5931 scope.go:122] "RemoveContainer" containerID="7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78" Nov 03 21:42:53.160831 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:53.160370 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78\": container with ID starting with 7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78 not found: ID does not exist" containerID="7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78" Nov 03 21:42:53.160831 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.160397 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78"} err="failed to get container status \"7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78\": rpc error: code = NotFound desc = could not find container \"7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78\": container with ID starting with 7a0a0ca14375ebc4c628257797ac33d8e1c6895a810534f1a90cdaef6df31d78 not found: ID does not exist" Nov 03 21:42:53.160831 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.160422 5931 scope.go:122] "RemoveContainer" containerID="d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce" Nov 03 21:42:53.171685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.171667 5931 scope.go:122] "RemoveContainer" containerID="d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce" Nov 03 21:42:53.171972 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:42:53.171942 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce\": container with ID starting with d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce not found: ID does not exist" containerID="d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce" Nov 03 21:42:53.172048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.171976 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce"} err="failed to get container status \"d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce\": rpc error: code = NotFound desc = could not find container \"d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce\": container with ID starting with d532a89929ed4de8a56a0e69c1e2af98b90a07fad45fe693487c4e7f354c84ce not found: ID does not exist" Nov 03 21:42:53.312121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.312041 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l"] Nov 03 21:42:53.360421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.360394 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1662df5-854b5fd75f-p577l"] Nov 03 21:42:53.436276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.436248 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn"] Nov 03 21:42:53.439998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.439971 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1662df5-5777b6f6b-gghdn"] Nov 03 21:42:53.792932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:53.792900 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:54.506044 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:54.506014 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" path="/var/lib/kubelet/pods/92614c7d-1f19-4a6e-95cb-5ba987bd3bb1/volumes" Nov 03 21:42:54.506407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:54.506391 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c8c7fd-bc3a-42d3-807c-f77238e0e732" path="/var/lib/kubelet/pods/c9c8c7fd-bc3a-42d3-807c-f77238e0e732/volumes" Nov 03 21:42:56.989897 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:56.989864 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt"] Nov 03 21:42:57.365909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:57.365861 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:57.596996 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:57.596956 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:59.195054 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:42:59.195021 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:01.990893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:01.990860 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" podUID="1f3189da-e24c-4663-90fb-2a934bda2de4" containerName="payload-pre-processing" containerID="cri-o://6a5a4a0d0b15cb81e27186401ef43236f6a2a3be8c727d9b0f051e77adc3e24d" gracePeriod=25 Nov 03 21:43:02.223777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.223745 5931 generic.go:356] "Generic (PLEG): container finished" podID="1f3189da-e24c-4663-90fb-2a934bda2de4" containerID="6a5a4a0d0b15cb81e27186401ef43236f6a2a3be8c727d9b0f051e77adc3e24d" exitCode=0 Nov 03 21:43:02.223947 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.223873 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" event={"ID":"1f3189da-e24c-4663-90fb-2a934bda2de4","Type":"ContainerDied","Data":"6a5a4a0d0b15cb81e27186401ef43236f6a2a3be8c727d9b0f051e77adc3e24d"} Nov 03 21:43:02.275683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.275299 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:43:02.338353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.338321 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3189da-e24c-4663-90fb-2a934bda2de4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3189da-e24c-4663-90fb-2a934bda2de4-plugins-config-volume\") pod \"1f3189da-e24c-4663-90fb-2a934bda2de4\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " Nov 03 21:43:02.338508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.338483 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3189da-e24c-4663-90fb-2a934bda2de4-kube-api-access-kg8f5\" (UniqueName: \"kubernetes.io/projected/1f3189da-e24c-4663-90fb-2a934bda2de4-kube-api-access-kg8f5\") pod \"1f3189da-e24c-4663-90fb-2a934bda2de4\" (UID: \"1f3189da-e24c-4663-90fb-2a934bda2de4\") " Nov 03 21:43:02.338776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.338750 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3189da-e24c-4663-90fb-2a934bda2de4-plugins-config-volume" pod "1f3189da-e24c-4663-90fb-2a934bda2de4" (UID: "1f3189da-e24c-4663-90fb-2a934bda2de4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:02.338947 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.338932 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3189da-e24c-4663-90fb-2a934bda2de4-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:02.340741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.340715 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3189da-e24c-4663-90fb-2a934bda2de4-kube-api-access-kg8f5" pod "1f3189da-e24c-4663-90fb-2a934bda2de4" (UID: "1f3189da-e24c-4663-90fb-2a934bda2de4"). InnerVolumeSpecName "kube-api-access-kg8f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:02.440500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:02.440464 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg8f5\" (UniqueName: \"kubernetes.io/projected/1f3189da-e24c-4663-90fb-2a934bda2de4-kube-api-access-kg8f5\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:03.068308 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:03.068274 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:43:03.232302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:03.232272 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" Nov 03 21:43:03.232452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:03.232305 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt" event={"ID":"1f3189da-e24c-4663-90fb-2a934bda2de4","Type":"ContainerDied","Data":"b1aaf5c9f0e48ebc6ccd666b06a74a05a83a7cc3e6375f5b57fe895bdf2fe548"} Nov 03 21:43:03.232452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:03.232368 5931 scope.go:122] "RemoveContainer" containerID="6a5a4a0d0b15cb81e27186401ef43236f6a2a3be8c727d9b0f051e77adc3e24d" Nov 03 21:43:03.395927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:03.395901 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt"] Nov 03 21:43:03.445453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:03.445425 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dad161-696458f7dc-8rqmt"] Nov 03 21:43:04.016481 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:04.016443 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:04.506110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:04.506079 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3189da-e24c-4663-90fb-2a934bda2de4" path="/var/lib/kubelet/pods/1f3189da-e24c-4663-90fb-2a934bda2de4/volumes" Nov 03 21:43:08.741197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:08.741165 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:13.419687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:13.419653 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:18.186835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:18.186805 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:19.364525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:19.364492 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb"] Nov 03 21:43:19.364931 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:19.364733 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" containerID="cri-o://f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a" gracePeriod=30 Nov 03 21:43:19.415647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:19.415578 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t"] Nov 03 21:43:19.415874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:19.415841 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" containerID="cri-o://5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7" gracePeriod=30 Nov 03 21:43:19.418912 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:19.418887 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6"] Nov 03 21:43:19.419071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:19.419053 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" containerID="cri-o://40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2" gracePeriod=30 Nov 03 21:43:22.017456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.017427 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z"] Nov 03 21:43:22.017868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.017646 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" podUID="8a114a05-b189-41b7-9735-809e1684cee0" containerName="maas-api" containerID="cri-o://fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90" gracePeriod=30 Nov 03 21:43:22.220341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.220305 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:22.293817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.293791 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:43:22.360491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.360466 5931 generic.go:356] "Generic (PLEG): container finished" podID="8a114a05-b189-41b7-9735-809e1684cee0" containerID="fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90" exitCode=0 Nov 03 21:43:22.360682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.360516 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" Nov 03 21:43:22.360682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.360642 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" event={"ID":"8a114a05-b189-41b7-9735-809e1684cee0","Type":"ContainerDied","Data":"fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90"} Nov 03 21:43:22.360773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.360697 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z" event={"ID":"8a114a05-b189-41b7-9735-809e1684cee0","Type":"ContainerDied","Data":"afe05772ee29b03d46613484d5edf3a333a79ba20b7532a978d6ec3dc18916aa"} Nov 03 21:43:22.360773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.360720 5931 scope.go:122] "RemoveContainer" containerID="fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90" Nov 03 21:43:22.361300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.361279 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a114a05-b189-41b7-9735-809e1684cee0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a114a05-b189-41b7-9735-809e1684cee0-openshift-service-ca\") pod \"8a114a05-b189-41b7-9735-809e1684cee0\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " Nov 03 21:43:22.361427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.361382 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-metrics-certs\") pod \"8a114a05-b189-41b7-9735-809e1684cee0\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " Nov 03 21:43:22.361497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.361424 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls\") pod \"8a114a05-b189-41b7-9735-809e1684cee0\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " Nov 03 21:43:22.361557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.361519 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a114a05-b189-41b7-9735-809e1684cee0-kube-api-access-df6n5\" (UniqueName: \"kubernetes.io/projected/8a114a05-b189-41b7-9735-809e1684cee0-kube-api-access-df6n5\") pod \"8a114a05-b189-41b7-9735-809e1684cee0\" (UID: \"8a114a05-b189-41b7-9735-809e1684cee0\") " Nov 03 21:43:22.361780 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.361728 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a114a05-b189-41b7-9735-809e1684cee0-openshift-service-ca" pod "8a114a05-b189-41b7-9735-809e1684cee0" (UID: "8a114a05-b189-41b7-9735-809e1684cee0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:22.362386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.362356 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a114a05-b189-41b7-9735-809e1684cee0-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:22.363775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.363754 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a114a05-b189-41b7-9735-809e1684cee0-kube-api-access-df6n5" pod "8a114a05-b189-41b7-9735-809e1684cee0" (UID: "8a114a05-b189-41b7-9735-809e1684cee0"). InnerVolumeSpecName "kube-api-access-df6n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:22.363875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.363846 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-metrics-certs" pod "8a114a05-b189-41b7-9735-809e1684cee0" (UID: "8a114a05-b189-41b7-9735-809e1684cee0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:22.363919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.363898 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls" pod "8a114a05-b189-41b7-9735-809e1684cee0" (UID: "8a114a05-b189-41b7-9735-809e1684cee0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:22.394029 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.393994 5931 scope.go:122] "RemoveContainer" containerID="fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90" Nov 03 21:43:22.394407 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:22.394367 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90\": container with ID starting with fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90 not found: ID does not exist" containerID="fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90" Nov 03 21:43:22.394484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.394402 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90"} err="failed to get container status \"fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90\": rpc error: code = NotFound desc = could not find container \"fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90\": container with ID starting with fea208fa26f08798b711c0cdb23f467e3161b61b8f9684d6a752afc12a62ef90 not found: ID does not exist" Nov 03 21:43:22.463758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.463723 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:22.463758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.463759 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a114a05-b189-41b7-9735-809e1684cee0-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:22.464300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.463774 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df6n5\" (UniqueName: \"kubernetes.io/projected/8a114a05-b189-41b7-9735-809e1684cee0-kube-api-access-df6n5\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:22.781323 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.781294 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z"] Nov 03 21:43:22.831913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:22.831880 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1662df5-6855c5b68b-fff8z"] Nov 03 21:43:23.803559 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:23.803503 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:24.034790 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.034750 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:24.294109 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.294081 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:24.507425 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.507398 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a114a05-b189-41b7-9735-809e1684cee0" path="/var/lib/kubelet/pods/8a114a05-b189-41b7-9735-809e1684cee0/volumes" Nov 03 21:43:24.706733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.706704 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:43:24.743453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.743433 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:43:24.746510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.746490 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:43:24.747156 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.747137 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:24.799491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799469 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-certs\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.799588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799527 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-podinfo\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.799588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799556 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-data\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.799588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799583 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-token\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-token\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.799743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799625 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-data\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.799743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799661 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-credential-socket\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.799743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799695 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-token\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-token\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.799893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799775 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-kube-api-access-7xcw7\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-kube-api-access-7xcw7\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.799893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799757 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-certs" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.799893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799804 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-kube-api-access-qtck7\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-kube-api-access-qtck7\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.799893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799824 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-data" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.799893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799832 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-podinfo\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.799893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799875 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-data" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.800256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799956 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-envoy\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.800256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.799999 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-socket\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.800256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800035 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-socket\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.800256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800072 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-kube-api-access-ck99j\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-kube-api-access-ck99j\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.800256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800108 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-credential-socket\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.800256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800199 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-envoy\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.800256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800225 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istiod-ca-cert\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800291 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-data\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800326 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-socket\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800273 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-credential-socket" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800374 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istiod-ca-cert\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800472 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-credential-socket\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800512 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-token\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-token\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800566 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-certs\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800632 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-podinfo\") pod \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\" (UID: \"e00308f0-47ba-4b9b-afb9-edcd6e994a4f\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800663 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-envoy\") pod \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\" (UID: \"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800691 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-certs\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.800812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800792 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istiod-ca-cert\") pod \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\" (UID: \"1756cafc-d3aa-4c56-a1cc-95006bf629c1\") " Nov 03 21:43:24.801375 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.800891 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istiod-ca-cert" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:24.801526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.801497 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.801594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.801532 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.801594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.801547 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.801594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.801560 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.801594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.801576 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.801958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.801936 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istiod-ca-cert" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:24.802183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802161 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-credential-socket" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.802274 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802215 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-podinfo" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:43:24.802341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802295 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-token" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:24.802492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802464 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-credential-socket" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.802596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802520 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-kube-api-access-7xcw7" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "kube-api-access-7xcw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:24.802710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802651 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-podinfo" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:43:24.802710 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802686 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-socket" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.802871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802851 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-socket" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.802928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.802879 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-token" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:24.803504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.803281 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-data" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.803504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.803305 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-envoy" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.803504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.803449 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-certs" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.803504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.803478 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-socket" pod "1756cafc-d3aa-4c56-a1cc-95006bf629c1" (UID: "1756cafc-d3aa-4c56-a1cc-95006bf629c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.804266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.803675 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istiod-ca-cert" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:24.804266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.803740 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-certs" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.804493 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.804473 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-kube-api-access-qtck7" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "kube-api-access-qtck7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:24.804791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.804754 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-token" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:24.804881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.804842 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-podinfo" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:43:24.805148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.805127 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-envoy" pod "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" (UID: "9b66ff23-cdbf-42e6-b567-ca2aba8e82f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.805253 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.805201 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-kube-api-access-ck99j" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "kube-api-access-ck99j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:24.805508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.805489 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-envoy" pod "e00308f0-47ba-4b9b-afb9-edcd6e994a4f" (UID: "e00308f0-47ba-4b9b-afb9-edcd6e994a4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:24.902465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902441 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902465 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902479 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902495 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xcw7\" (UniqueName: \"kubernetes.io/projected/1756cafc-d3aa-4c56-a1cc-95006bf629c1-kube-api-access-7xcw7\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902507 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtck7\" (UniqueName: \"kubernetes.io/projected/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-kube-api-access-qtck7\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902519 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902531 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902542 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902555 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902567 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck99j\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-kube-api-access-ck99j\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902579 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902592 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902625 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902640 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902653 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902666 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902679 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.902688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902692 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.903193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902704 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00308f0-47ba-4b9b-afb9-edcd6e994a4f-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.903193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902719 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.903193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902733 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1756cafc-d3aa-4c56-a1cc-95006bf629c1-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:24.903193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:24.902745 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1756cafc-d3aa-4c56-a1cc-95006bf629c1-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:25.384700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.384667 5931 generic.go:356] "Generic (PLEG): container finished" podID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerID="40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2" exitCode=0 Nov 03 21:43:25.384846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.384720 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" Nov 03 21:43:25.384846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.384748 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" event={"ID":"e00308f0-47ba-4b9b-afb9-edcd6e994a4f","Type":"ContainerDied","Data":"40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2"} Nov 03 21:43:25.384846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.384804 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6" event={"ID":"e00308f0-47ba-4b9b-afb9-edcd6e994a4f","Type":"ContainerDied","Data":"60e8878d51af2f0b39acaa8383a6eac3ad8f2d8656220faabc2470bfe6bb753e"} Nov 03 21:43:25.384846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.384828 5931 scope.go:122] "RemoveContainer" containerID="40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2" Nov 03 21:43:25.391135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.391112 5931 generic.go:356] "Generic (PLEG): container finished" podID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerID="f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a" exitCode=0 Nov 03 21:43:25.391258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.391159 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" Nov 03 21:43:25.391258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.391188 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" event={"ID":"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2","Type":"ContainerDied","Data":"f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a"} Nov 03 21:43:25.391258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.391229 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb" event={"ID":"9b66ff23-cdbf-42e6-b567-ca2aba8e82f2","Type":"ContainerDied","Data":"ff3e043ae15be616477f216b1ebc02c663be46acf248fbd37eee701b3d3d314a"} Nov 03 21:43:25.392669 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.392645 5931 generic.go:356] "Generic (PLEG): container finished" podID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerID="5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7" exitCode=0 Nov 03 21:43:25.392771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.392684 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" Nov 03 21:43:25.392771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.392728 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" event={"ID":"1756cafc-d3aa-4c56-a1cc-95006bf629c1","Type":"ContainerDied","Data":"5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7"} Nov 03 21:43:25.392862 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.392776 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t" event={"ID":"1756cafc-d3aa-4c56-a1cc-95006bf629c1","Type":"ContainerDied","Data":"9ea9d2b8a3dfaafc220b1c476c829f594fd429b5c14af41064923256bef023f3"} Nov 03 21:43:25.413501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.413484 5931 scope.go:122] "RemoveContainer" containerID="40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2" Nov 03 21:43:25.413769 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:25.413751 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2\": container with ID starting with 40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2 not found: ID does not exist" containerID="40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2" Nov 03 21:43:25.413861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.413775 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2"} err="failed to get container status \"40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2\": rpc error: code = NotFound desc = could not find container \"40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2\": container with ID starting with 40914c90ece766a2c3130faddac9b2f644aac5e6a79d53103cf3007e4903e7c2 not found: ID does not exist" Nov 03 21:43:25.413861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.413790 5931 scope.go:122] "RemoveContainer" containerID="f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a" Nov 03 21:43:25.423136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.423118 5931 scope.go:122] "RemoveContainer" containerID="f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a" Nov 03 21:43:25.423395 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:25.423378 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a\": container with ID starting with f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a not found: ID does not exist" containerID="f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a" Nov 03 21:43:25.423443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.423401 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a"} err="failed to get container status \"f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a\": rpc error: code = NotFound desc = could not find container \"f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a\": container with ID starting with f47846a546b1a63082ce6ebcbcbab980b1fb591c572e50fb796b16073439888a not found: ID does not exist" Nov 03 21:43:25.423443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.423418 5931 scope.go:122] "RemoveContainer" containerID="5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7" Nov 03 21:43:25.431938 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.431922 5931 scope.go:122] "RemoveContainer" containerID="5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7" Nov 03 21:43:25.432195 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:25.432175 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7\": container with ID starting with 5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7 not found: ID does not exist" containerID="5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7" Nov 03 21:43:25.432238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.432203 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7"} err="failed to get container status \"5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7\": rpc error: code = NotFound desc = could not find container \"5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7\": container with ID starting with 5e2c28016dab922186f12b046f4c292b702ac4584e01c59d40916ca1d422e6a7 not found: ID does not exist" Nov 03 21:43:25.548200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.548175 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6"] Nov 03 21:43:25.553979 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.553955 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8cfq6"] Nov 03 21:43:25.662426 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.662362 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb"] Nov 03 21:43:25.666329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.666310 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9xjb"] Nov 03 21:43:25.774087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.774056 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t"] Nov 03 21:43:25.779565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:25.779539 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-7pj5t"] Nov 03 21:43:26.505945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:26.505914 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" path="/var/lib/kubelet/pods/1756cafc-d3aa-4c56-a1cc-95006bf629c1/volumes" Nov 03 21:43:26.506356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:26.506338 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" path="/var/lib/kubelet/pods/9b66ff23-cdbf-42e6-b567-ca2aba8e82f2/volumes" Nov 03 21:43:26.506761 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:26.506749 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" path="/var/lib/kubelet/pods/e00308f0-47ba-4b9b-afb9-edcd6e994a4f/volumes" Nov 03 21:43:29.045941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:29.045719 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:29.603817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:29.603790 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w"] Nov 03 21:43:29.798126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:29.798093 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:30.444592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:30.444553 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:33.086970 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:33.086931 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:43:34.280645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:34.280586 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:39.409672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:39.409596 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:41.713104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:41.713064 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:43.135905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:43.135873 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f"] Nov 03 21:43:43.662063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:43.662030 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:43.888499 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:43.888467 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:44.147228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:44.147189 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:44.623709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:44.623675 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" podUID="a9234ec9-004d-4ffe-8001-9118274d8894" containerName="main" containerID="cri-o://2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e" gracePeriod=45 Nov 03 21:43:44.916439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:44.916373 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:43:45.015004 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.014968 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-model-cache\") pod \"a9234ec9-004d-4ffe-8001-9118274d8894\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " Nov 03 21:43:45.015190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015066 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-dshm\") pod \"a9234ec9-004d-4ffe-8001-9118274d8894\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " Nov 03 21:43:45.015190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015093 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9234ec9-004d-4ffe-8001-9118274d8894-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9234ec9-004d-4ffe-8001-9118274d8894-tls-certs\") pod \"a9234ec9-004d-4ffe-8001-9118274d8894\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " Nov 03 21:43:45.015190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015122 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-tmp-dir\") pod \"a9234ec9-004d-4ffe-8001-9118274d8894\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " Nov 03 21:43:45.015371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015198 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9234ec9-004d-4ffe-8001-9118274d8894-kube-api-access-g8r8d\" (UniqueName: \"kubernetes.io/projected/a9234ec9-004d-4ffe-8001-9118274d8894-kube-api-access-g8r8d\") pod \"a9234ec9-004d-4ffe-8001-9118274d8894\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " Nov 03 21:43:45.015371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015231 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-home\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-home\") pod \"a9234ec9-004d-4ffe-8001-9118274d8894\" (UID: \"a9234ec9-004d-4ffe-8001-9118274d8894\") " Nov 03 21:43:45.015371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015229 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-model-cache" pod "a9234ec9-004d-4ffe-8001-9118274d8894" (UID: "a9234ec9-004d-4ffe-8001-9118274d8894"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:45.015527 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015430 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-tmp-dir" pod "a9234ec9-004d-4ffe-8001-9118274d8894" (UID: "a9234ec9-004d-4ffe-8001-9118274d8894"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:45.015527 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.015512 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-home" pod "a9234ec9-004d-4ffe-8001-9118274d8894" (UID: "a9234ec9-004d-4ffe-8001-9118274d8894"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:45.016515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.016491 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:45.016670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.016602 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:45.016670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.016643 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:45.017825 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.017801 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9234ec9-004d-4ffe-8001-9118274d8894-tls-certs" pod "a9234ec9-004d-4ffe-8001-9118274d8894" (UID: "a9234ec9-004d-4ffe-8001-9118274d8894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:45.018041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.018019 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9234ec9-004d-4ffe-8001-9118274d8894-kube-api-access-g8r8d" pod "a9234ec9-004d-4ffe-8001-9118274d8894" (UID: "a9234ec9-004d-4ffe-8001-9118274d8894"). InnerVolumeSpecName "kube-api-access-g8r8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:45.018103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.018038 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-dshm" pod "a9234ec9-004d-4ffe-8001-9118274d8894" (UID: "a9234ec9-004d-4ffe-8001-9118274d8894"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:45.117987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.117950 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8r8d\" (UniqueName: \"kubernetes.io/projected/a9234ec9-004d-4ffe-8001-9118274d8894-kube-api-access-g8r8d\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:45.117987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.117976 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9234ec9-004d-4ffe-8001-9118274d8894-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:45.117987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.117985 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9234ec9-004d-4ffe-8001-9118274d8894-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:45.289489 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.289380 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:45.519859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.519817 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:45.550555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.550485 5931 generic.go:356] "Generic (PLEG): container finished" podID="a9234ec9-004d-4ffe-8001-9118274d8894" containerID="2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e" exitCode=0 Nov 03 21:43:45.550555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.550518 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" Nov 03 21:43:45.550748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.550570 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" event={"ID":"a9234ec9-004d-4ffe-8001-9118274d8894","Type":"ContainerDied","Data":"2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e"} Nov 03 21:43:45.550748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.550604 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w" event={"ID":"a9234ec9-004d-4ffe-8001-9118274d8894","Type":"ContainerDied","Data":"da2517f08bfd6ab82206508116254fc724e99aeca1900b827168a88a2dd30198"} Nov 03 21:43:45.550748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.550633 5931 scope.go:122] "RemoveContainer" containerID="2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e" Nov 03 21:43:45.564594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.564566 5931 scope.go:122] "RemoveContainer" containerID="2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e" Nov 03 21:43:45.565272 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:45.565248 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e\": container with ID starting with 2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e not found: ID does not exist" containerID="2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e" Nov 03 21:43:45.565360 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.565282 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e"} err="failed to get container status \"2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e\": rpc error: code = NotFound desc = could not find container \"2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e\": container with ID starting with 2897d764f09f279e309e2541ed127ad2ded67f491f8048d1a18f5bc14bbc943e not found: ID does not exist" Nov 03 21:43:45.714757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.714732 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w"] Nov 03 21:43:45.763750 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:45.763731 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c1662df5/e2e-lifecycle-model-c1662df5-kserve-647cf5cb8f-fhk6w"] Nov 03 21:43:46.506190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:46.506156 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9234ec9-004d-4ffe-8001-9118274d8894" path="/var/lib/kubelet/pods/a9234ec9-004d-4ffe-8001-9118274d8894/volumes" Nov 03 21:43:48.086759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.086722 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt"] Nov 03 21:43:48.087136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.086915 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" podUID="b876b70b-abc3-49fa-b206-70bc6ccdfc76" containerName="maas-api" containerID="cri-o://726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5" gracePeriod=30 Nov 03 21:43:48.137006 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.136977 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" podUID="664373db-8da4-482d-b5e3-70fad0c56bf8" containerName="payload-processing" containerID="cri-o://f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198" gracePeriod=25 Nov 03 21:43:48.279273 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.279247 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:48.432784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.432759 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:43:48.481759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.481728 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:43:48.558073 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.558044 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-metrics-certs\") pod \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " Nov 03 21:43:48.558267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.558117 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls\") pod \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " Nov 03 21:43:48.558267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.558223 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b876b70b-abc3-49fa-b206-70bc6ccdfc76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b876b70b-abc3-49fa-b206-70bc6ccdfc76-openshift-service-ca\") pod \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " Nov 03 21:43:48.558384 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.558283 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b876b70b-abc3-49fa-b206-70bc6ccdfc76-kube-api-access-j4qd9\" (UniqueName: \"kubernetes.io/projected/b876b70b-abc3-49fa-b206-70bc6ccdfc76-kube-api-access-j4qd9\") pod \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\" (UID: \"b876b70b-abc3-49fa-b206-70bc6ccdfc76\") " Nov 03 21:43:48.560358 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.560328 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b876b70b-abc3-49fa-b206-70bc6ccdfc76-openshift-service-ca" pod "b876b70b-abc3-49fa-b206-70bc6ccdfc76" (UID: "b876b70b-abc3-49fa-b206-70bc6ccdfc76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:48.560972 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.560946 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-metrics-certs" pod "b876b70b-abc3-49fa-b206-70bc6ccdfc76" (UID: "b876b70b-abc3-49fa-b206-70bc6ccdfc76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:48.561202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.561180 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls" pod "b876b70b-abc3-49fa-b206-70bc6ccdfc76" (UID: "b876b70b-abc3-49fa-b206-70bc6ccdfc76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:48.574159 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.574131 5931 generic.go:356] "Generic (PLEG): container finished" podID="b876b70b-abc3-49fa-b206-70bc6ccdfc76" containerID="726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5" exitCode=0 Nov 03 21:43:48.574288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.574162 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" event={"ID":"b876b70b-abc3-49fa-b206-70bc6ccdfc76","Type":"ContainerDied","Data":"726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5"} Nov 03 21:43:48.574288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.574181 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" Nov 03 21:43:48.574288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.574233 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt" event={"ID":"b876b70b-abc3-49fa-b206-70bc6ccdfc76","Type":"ContainerDied","Data":"008e737bff6252e356b4f2f96ba76c1b4446b4dbe8cfc852949ed63fca7ac123"} Nov 03 21:43:48.574288 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.574265 5931 scope.go:122] "RemoveContainer" containerID="726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5" Nov 03 21:43:48.576472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.576451 5931 generic.go:356] "Generic (PLEG): container finished" podID="664373db-8da4-482d-b5e3-70fad0c56bf8" containerID="f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198" exitCode=0 Nov 03 21:43:48.576602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.576483 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" event={"ID":"664373db-8da4-482d-b5e3-70fad0c56bf8","Type":"ContainerDied","Data":"f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198"} Nov 03 21:43:48.576602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.576499 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" Nov 03 21:43:48.576602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.576533 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f" event={"ID":"664373db-8da4-482d-b5e3-70fad0c56bf8","Type":"ContainerDied","Data":"bd0b4004b86a3385d22f0e78cd3a1c26c93da6ad46654a59b21158910c09ab2d"} Nov 03 21:43:48.580974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.580953 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b876b70b-abc3-49fa-b206-70bc6ccdfc76-kube-api-access-j4qd9" pod "b876b70b-abc3-49fa-b206-70bc6ccdfc76" (UID: "b876b70b-abc3-49fa-b206-70bc6ccdfc76"). InnerVolumeSpecName "kube-api-access-j4qd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:48.584833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.584605 5931 scope.go:122] "RemoveContainer" containerID="726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5" Nov 03 21:43:48.584979 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:48.584952 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5\": container with ID starting with 726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5 not found: ID does not exist" containerID="726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5" Nov 03 21:43:48.585075 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.584984 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5"} err="failed to get container status \"726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5\": rpc error: code = NotFound desc = could not find container \"726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5\": container with ID starting with 726fed9d17d5a758aa9fa5ec34809e4951b262f4050f6665db2ad61ac695f0f5 not found: ID does not exist" Nov 03 21:43:48.585075 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.585008 5931 scope.go:122] "RemoveContainer" containerID="f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198" Nov 03 21:43:48.594654 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.594597 5931 scope.go:122] "RemoveContainer" containerID="f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198" Nov 03 21:43:48.594975 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:43:48.594949 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198\": container with ID starting with f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198 not found: ID does not exist" containerID="f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198" Nov 03 21:43:48.595057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.594985 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198"} err="failed to get container status \"f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198\": rpc error: code = NotFound desc = could not find container \"f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198\": container with ID starting with f1777e5f347aef7957a3ac865f26c3a6bc525a7ce5c47f6f0f00847f4661f198 not found: ID does not exist" Nov 03 21:43:48.660302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.660212 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664373db-8da4-482d-b5e3-70fad0c56bf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664373db-8da4-482d-b5e3-70fad0c56bf8-plugins-config-volume\") pod \"664373db-8da4-482d-b5e3-70fad0c56bf8\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " Nov 03 21:43:48.660457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.660404 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664373db-8da4-482d-b5e3-70fad0c56bf8-kube-api-access-xcr5n\" (UniqueName: \"kubernetes.io/projected/664373db-8da4-482d-b5e3-70fad0c56bf8-kube-api-access-xcr5n\") pod \"664373db-8da4-482d-b5e3-70fad0c56bf8\" (UID: \"664373db-8da4-482d-b5e3-70fad0c56bf8\") " Nov 03 21:43:48.660771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.660709 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664373db-8da4-482d-b5e3-70fad0c56bf8-plugins-config-volume" pod "664373db-8da4-482d-b5e3-70fad0c56bf8" (UID: "664373db-8da4-482d-b5e3-70fad0c56bf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:48.662227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.662196 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664373db-8da4-482d-b5e3-70fad0c56bf8-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:48.662340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.662230 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:48.662340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.662247 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b876b70b-abc3-49fa-b206-70bc6ccdfc76-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:48.662340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.662260 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b876b70b-abc3-49fa-b206-70bc6ccdfc76-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:48.662340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.662273 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4qd9\" (UniqueName: \"kubernetes.io/projected/b876b70b-abc3-49fa-b206-70bc6ccdfc76-kube-api-access-j4qd9\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:48.664638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.664576 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664373db-8da4-482d-b5e3-70fad0c56bf8-kube-api-access-xcr5n" pod "664373db-8da4-482d-b5e3-70fad0c56bf8" (UID: "664373db-8da4-482d-b5e3-70fad0c56bf8"). InnerVolumeSpecName "kube-api-access-xcr5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:48.763027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.762998 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcr5n\" (UniqueName: \"kubernetes.io/projected/664373db-8da4-482d-b5e3-70fad0c56bf8-kube-api-access-xcr5n\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:43:49.004263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:48.997302 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f"] Nov 03 21:43:49.004700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:49.004675 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fff189-7d6bccf6c8-8qf9f"] Nov 03 21:43:49.125500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:49.125466 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt"] Nov 03 21:43:49.128853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:49.128824 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30f143-67db4ffb9f-4p8wt"] Nov 03 21:43:49.921235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:49.921192 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:50.143262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:50.143235 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:50.373119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:50.373093 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:50.505979 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:50.505945 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664373db-8da4-482d-b5e3-70fad0c56bf8" path="/var/lib/kubelet/pods/664373db-8da4-482d-b5e3-70fad0c56bf8/volumes" Nov 03 21:43:50.506356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:50.506343 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b876b70b-abc3-49fa-b206-70bc6ccdfc76" path="/var/lib/kubelet/pods/b876b70b-abc3-49fa-b206-70bc6ccdfc76/volumes" Nov 03 21:43:50.861868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:50.861841 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:55.236035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:55.236001 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:56.857874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:56.857842 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:57.139281 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:57.139199 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:57.373855 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:43:57.373824 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:00.297109 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:00.297069 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:03.068006 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:44:03.067971 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:44:05.750709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:05.750677 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:06.340166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:06.340137 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:06.610983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:06.610905 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:09.494479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:09.494440 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59"] Nov 03 21:44:09.612818 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:09.612787 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q"] Nov 03 21:44:10.018138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:10.018108 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:10.256457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:10.256430 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:10.584741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:10.584718 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:12.363165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.363097 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:12.585216 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.585185 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf"] Nov 03 21:44:12.585390 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.585353 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" containerID="cri-o://1132d54da47b7fe44c8e4ae442a4401fc9e4226e11f6ab74c68281de13956684" gracePeriod=30 Nov 03 21:44:12.599885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.599856 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc"] Nov 03 21:44:12.600085 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.600063 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" containerID="cri-o://105a9c5ecdbc69a95b18261a418101fdfdf0f85706830f1800df687743d673db" gracePeriod=30 Nov 03 21:44:12.605567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.605547 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:12.621967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.621905 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95"] Nov 03 21:44:12.622120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.622099 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" containerID="cri-o://3526c912c77fdf6c2f041c19530199818951d55850aac26d61e84f3122db4557" gracePeriod=30 Nov 03 21:44:12.753842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.753812 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:44:12.753977 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.753893 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:44:12.865635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:12.847988 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:13.006685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.006569 5931 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:44:13.006685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.006639 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:44:13.505081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.505044 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:13.936599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.936561 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h"] Nov 03 21:44:13.937139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937123 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" Nov 03 21:44:13.937139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937139 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937153 5931 cpu_manager.go:414] "containerMap: removing container" podUID="92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" containerName="payload-pre-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937159 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" containerName="payload-pre-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937170 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8a114a05-b189-41b7-9735-809e1684cee0" containerName="maas-api" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937174 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a114a05-b189-41b7-9735-809e1684cee0" containerName="maas-api" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937186 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c9c8c7fd-bc3a-42d3-807c-f77238e0e732" containerName="payload-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937191 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c8c7fd-bc3a-42d3-807c-f77238e0e732" containerName="payload-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937199 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b876b70b-abc3-49fa-b206-70bc6ccdfc76" containerName="maas-api" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937204 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b876b70b-abc3-49fa-b206-70bc6ccdfc76" containerName="maas-api" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937214 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937218 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937226 5931 cpu_manager.go:414] "containerMap: removing container" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937232 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937240 5931 cpu_manager.go:414] "containerMap: removing container" podUID="cddceccb-7d62-43b5-abad-da056ea7fc0c" containerName="payload-pre-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937244 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cddceccb-7d62-43b5-abad-da056ea7fc0c" containerName="payload-pre-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937250 5931 cpu_manager.go:414] "containerMap: removing container" podUID="664373db-8da4-482d-b5e3-70fad0c56bf8" containerName="payload-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937254 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664373db-8da4-482d-b5e3-70fad0c56bf8" containerName="payload-processing" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937266 5931 cpu_manager.go:414] "containerMap: removing container" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" Nov 03 21:44:13.937263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937271 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937278 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937282 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937289 5931 cpu_manager.go:414] "containerMap: removing container" podUID="a9234ec9-004d-4ffe-8001-9118274d8894" containerName="main" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937294 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9234ec9-004d-4ffe-8001-9118274d8894" containerName="main" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937302 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1f3189da-e24c-4663-90fb-2a934bda2de4" containerName="payload-pre-processing" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937306 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3189da-e24c-4663-90fb-2a934bda2de4" containerName="payload-pre-processing" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937314 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937318 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937325 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937329 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937337 5931 cpu_manager.go:414] "containerMap: removing container" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937340 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937347 5931 cpu_manager.go:414] "containerMap: removing container" podUID="5f243fff-cdba-45cd-8096-397653259359" containerName="maas-api" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937351 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f243fff-cdba-45cd-8096-397653259359" containerName="maas-api" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937429 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937436 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="92614c7d-1f19-4a6e-95cb-5ba987bd3bb1" containerName="payload-pre-processing" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937443 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a114a05-b189-41b7-9735-809e1684cee0" containerName="maas-api" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937450 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c8c7fd-bc3a-42d3-807c-f77238e0e732" containerName="payload-processing" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937458 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b876b70b-abc3-49fa-b206-70bc6ccdfc76" containerName="maas-api" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937464 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4287f3-df53-47db-874c-3e8d3bfd92f6" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937470 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b66ff23-cdbf-42e6-b567-ca2aba8e82f2" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937474 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cddceccb-7d62-43b5-abad-da056ea7fc0c" containerName="payload-pre-processing" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937480 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="664373db-8da4-482d-b5e3-70fad0c56bf8" containerName="payload-processing" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937486 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="de70907b-7e1b-462e-beb9-50b66b931bf9" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937491 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c141d35b-42f3-42c7-9877-4e918a21b4dc" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937496 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9234ec9-004d-4ffe-8001-9118274d8894" containerName="main" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937502 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3189da-e24c-4663-90fb-2a934bda2de4" containerName="payload-pre-processing" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937507 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c895147-e359-4fce-ab8a-f58a70f58b82" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937512 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1756cafc-d3aa-4c56-a1cc-95006bf629c1" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937518 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00308f0-47ba-4b9b-afb9-edcd6e994a4f" containerName="istio-proxy" Nov 03 21:44:13.937814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.937522 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f243fff-cdba-45cd-8096-397653259359" containerName="maas-api" Nov 03 21:44:13.940967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:13.940951 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021041 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-data\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021138 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-socket\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021173 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-podinfo\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021227 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-envoy\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021253 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-token\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021336 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892329ca-8914-4602-ae6a-88a3d8666cdf-istiod-ca-cert\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021407 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-certs\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021445 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdq4\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-kube-api-access-gtdq4\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.021478 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-credential-socket\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.030637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.026675 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h"] Nov 03 21:44:14.052433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.052398 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a1931b-openshift-default-dockercfg-tw2kw\"" Nov 03 21:44:14.122738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.122709 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-envoy\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.122738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.122740 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-token\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.122824 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892329ca-8914-4602-ae6a-88a3d8666cdf-istiod-ca-cert\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.122894 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-certs\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.122930 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdq4\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-kube-api-access-gtdq4\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.122960 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-credential-socket\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123008 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-data\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123096 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-socket\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123126 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-podinfo\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123297 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-certs\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123498 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-socket\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123718 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123672 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-credential-socket\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123765 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-data\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.123898 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.123878 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892329ca-8914-4602-ae6a-88a3d8666cdf-istiod-ca-cert\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.125980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.125950 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-envoy\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.126147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.126131 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-podinfo\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.126654 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.126577 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:14.181004 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.180974 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdq4\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-kube-api-access-gtdq4\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.181564 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.181544 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-token\") pod \"e2e-api-a-a1931b-openshift-default-679465bf47-4w76h\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.254532 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.254457 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:14.360407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.360378 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:14.494993 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.494954 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" podUID="4ca62afe-8252-49db-99f9-d591f81b4392" containerName="payload-processing" containerID="cri-o://a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe" gracePeriod=25 Nov 03 21:44:14.613207 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.613154 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" podUID="7b44aee0-da1d-4e4e-a775-0842abb2ae51" containerName="payload-pre-processing" containerID="cri-o://51f634f8ae36f4ab7f38d24ca1ccffd1f558f825c67d6291079a16ae14b25f22" gracePeriod=25 Nov 03 21:44:14.688089 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.686463 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h"] Nov 03 21:44:14.688089 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:44:14.687714 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892329ca_8914_4602_ae6a_88a3d8666cdf.slice/crio-73e9de3d9b6fcb7c80dacd2764cc37220f8fb0d6bc2316d4f26d2120a59f3b01 WatchSource:0}: Error finding container 73e9de3d9b6fcb7c80dacd2764cc37220f8fb0d6bc2316d4f26d2120a59f3b01: Status 404 returned error can't find the container with id 73e9de3d9b6fcb7c80dacd2764cc37220f8fb0d6bc2316d4f26d2120a59f3b01 Nov 03 21:44:14.718896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.718861 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:44:14.718999 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.718944 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:44:14.718999 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.718988 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:44:14.741768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.741749 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:44:14.756127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.756068 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:14.758356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.758332 5931 generic.go:356] "Generic (PLEG): container finished" podID="4ca62afe-8252-49db-99f9-d591f81b4392" containerID="a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe" exitCode=0 Nov 03 21:44:14.758468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.758451 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" Nov 03 21:44:14.758534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.758479 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" event={"ID":"4ca62afe-8252-49db-99f9-d591f81b4392","Type":"ContainerDied","Data":"a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe"} Nov 03 21:44:14.758602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.758536 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59" event={"ID":"4ca62afe-8252-49db-99f9-d591f81b4392","Type":"ContainerDied","Data":"dc20e30906953514ec77ee5c93ad78b5d48c12e0714f2019bc8e4588bc79cba1"} Nov 03 21:44:14.758602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.758573 5931 scope.go:122] "RemoveContainer" containerID="a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe" Nov 03 21:44:14.760411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.760388 5931 generic.go:356] "Generic (PLEG): container finished" podID="7b44aee0-da1d-4e4e-a775-0842abb2ae51" containerID="51f634f8ae36f4ab7f38d24ca1ccffd1f558f825c67d6291079a16ae14b25f22" exitCode=0 Nov 03 21:44:14.760494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.760478 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" event={"ID":"7b44aee0-da1d-4e4e-a775-0842abb2ae51","Type":"ContainerDied","Data":"51f634f8ae36f4ab7f38d24ca1ccffd1f558f825c67d6291079a16ae14b25f22"} Nov 03 21:44:14.761813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.761793 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" event={"ID":"892329ca-8914-4602-ae6a-88a3d8666cdf","Type":"ContainerStarted","Data":"73e9de3d9b6fcb7c80dacd2764cc37220f8fb0d6bc2316d4f26d2120a59f3b01"} Nov 03 21:44:14.779359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.779331 5931 scope.go:122] "RemoveContainer" containerID="a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe" Nov 03 21:44:14.779777 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:44:14.779754 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe\": container with ID starting with a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe not found: ID does not exist" containerID="a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe" Nov 03 21:44:14.779864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.779788 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe"} err="failed to get container status \"a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe\": rpc error: code = NotFound desc = could not find container \"a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe\": container with ID starting with a913aadf1ea2a98300ccd0702444378862ef9204576e8b06e337a2b9620069fe not found: ID does not exist" Nov 03 21:44:14.838636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.838309 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca62afe-8252-49db-99f9-d591f81b4392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca62afe-8252-49db-99f9-d591f81b4392-plugins-config-volume\") pod \"4ca62afe-8252-49db-99f9-d591f81b4392\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " Nov 03 21:44:14.838636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.838491 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca62afe-8252-49db-99f9-d591f81b4392-kube-api-access-ttd5k\" (UniqueName: \"kubernetes.io/projected/4ca62afe-8252-49db-99f9-d591f81b4392-kube-api-access-ttd5k\") pod \"4ca62afe-8252-49db-99f9-d591f81b4392\" (UID: \"4ca62afe-8252-49db-99f9-d591f81b4392\") " Nov 03 21:44:14.839268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.839224 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca62afe-8252-49db-99f9-d591f81b4392-plugins-config-volume" pod "4ca62afe-8252-49db-99f9-d591f81b4392" (UID: "4ca62afe-8252-49db-99f9-d591f81b4392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:14.840244 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.840211 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca62afe-8252-49db-99f9-d591f81b4392-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:14.873867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.873825 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca62afe-8252-49db-99f9-d591f81b4392-kube-api-access-ttd5k" pod "4ca62afe-8252-49db-99f9-d591f81b4392" (UID: "4ca62afe-8252-49db-99f9-d591f81b4392"). InnerVolumeSpecName "kube-api-access-ttd5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:14.890209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.890181 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:44:14.941809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:14.941783 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttd5k\" (UniqueName: \"kubernetes.io/projected/4ca62afe-8252-49db-99f9-d591f81b4392-kube-api-access-ttd5k\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:15.042671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.042580 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b44aee0-da1d-4e4e-a775-0842abb2ae51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b44aee0-da1d-4e4e-a775-0842abb2ae51-plugins-config-volume\") pod \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " Nov 03 21:44:15.042835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.042709 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b44aee0-da1d-4e4e-a775-0842abb2ae51-kube-api-access-2jbhh\" (UniqueName: \"kubernetes.io/projected/7b44aee0-da1d-4e4e-a775-0842abb2ae51-kube-api-access-2jbhh\") pod \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\" (UID: \"7b44aee0-da1d-4e4e-a775-0842abb2ae51\") " Nov 03 21:44:15.043027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.042998 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b44aee0-da1d-4e4e-a775-0842abb2ae51-plugins-config-volume" pod "7b44aee0-da1d-4e4e-a775-0842abb2ae51" (UID: "7b44aee0-da1d-4e4e-a775-0842abb2ae51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:15.043472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.043437 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b44aee0-da1d-4e4e-a775-0842abb2ae51-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:15.045452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.045423 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b44aee0-da1d-4e4e-a775-0842abb2ae51-kube-api-access-2jbhh" pod "7b44aee0-da1d-4e4e-a775-0842abb2ae51" (UID: "7b44aee0-da1d-4e4e-a775-0842abb2ae51"). InnerVolumeSpecName "kube-api-access-2jbhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:15.144246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.144217 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jbhh\" (UniqueName: \"kubernetes.io/projected/7b44aee0-da1d-4e4e-a775-0842abb2ae51-kube-api-access-2jbhh\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:15.180122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.180090 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59"] Nov 03 21:44:15.184134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.184102 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-85d7ff-7fb589cf76-zkx59"] Nov 03 21:44:15.770421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.770393 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" Nov 03 21:44:15.770883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.770394 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q" event={"ID":"7b44aee0-da1d-4e4e-a775-0842abb2ae51","Type":"ContainerDied","Data":"a866536d493b5a3aa07c6a99e1f0d698f780684961d9edaaf12546b8648a56b7"} Nov 03 21:44:15.770883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.770535 5931 scope.go:122] "RemoveContainer" containerID="51f634f8ae36f4ab7f38d24ca1ccffd1f558f825c67d6291079a16ae14b25f22" Nov 03 21:44:15.772271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.772248 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" event={"ID":"892329ca-8914-4602-ae6a-88a3d8666cdf","Type":"ContainerStarted","Data":"a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e"} Nov 03 21:44:15.932221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.932197 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q"] Nov 03 21:44:15.981155 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:15.981128 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-85d7ff-86dff56775-8rb2q"] Nov 03 21:44:16.093878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.093840 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podStartSLOduration=3.093828213 podStartE2EDuration="3.093828213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:16.09122559 +0000 UTC m=+1723.702073405" watchObservedRunningTime="2026-11-03 21:44:16.093828213 +0000 UTC m=+1723.704676027" Nov 03 21:44:16.255287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.255251 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:16.260970 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.260944 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:16.352913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.352822 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:16.506248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.506211 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca62afe-8252-49db-99f9-d591f81b4392" path="/var/lib/kubelet/pods/4ca62afe-8252-49db-99f9-d591f81b4392/volumes" Nov 03 21:44:16.506679 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.506660 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b44aee0-da1d-4e4e-a775-0842abb2ae51" path="/var/lib/kubelet/pods/7b44aee0-da1d-4e4e-a775-0842abb2ae51/volumes" Nov 03 21:44:16.583920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.583876 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:16.780350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.780275 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:16.781126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.781108 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:44:16.811218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:16.811136 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:17.295794 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.295755 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:17.791409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.788889 5931 generic.go:356] "Generic (PLEG): container finished" podID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerID="1132d54da47b7fe44c8e4ae442a4401fc9e4226e11f6ab74c68281de13956684" exitCode=0 Nov 03 21:44:17.791409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.789002 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" event={"ID":"e8efb056-b5a5-4098-b8ea-dac1bc7f1522","Type":"ContainerDied","Data":"1132d54da47b7fe44c8e4ae442a4401fc9e4226e11f6ab74c68281de13956684"} Nov 03 21:44:17.800058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.799238 5931 generic.go:356] "Generic (PLEG): container finished" podID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerID="3526c912c77fdf6c2f041c19530199818951d55850aac26d61e84f3122db4557" exitCode=0 Nov 03 21:44:17.800058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.799341 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" event={"ID":"b9931e2f-5857-4964-b99b-fe8eb3b8a281","Type":"ContainerDied","Data":"3526c912c77fdf6c2f041c19530199818951d55850aac26d61e84f3122db4557"} Nov 03 21:44:17.801922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.801880 5931 generic.go:356] "Generic (PLEG): container finished" podID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerID="105a9c5ecdbc69a95b18261a418101fdfdf0f85706830f1800df687743d673db" exitCode=0 Nov 03 21:44:17.802022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.801944 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" event={"ID":"6eedfea8-2d8c-4598-badd-d1063cc88c09","Type":"ContainerDied","Data":"105a9c5ecdbc69a95b18261a418101fdfdf0f85706830f1800df687743d673db"} Nov 03 21:44:17.894051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.894028 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:44:17.935643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.935606 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:44:17.938696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.938678 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:44:17.976849 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.976829 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-socket\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.976967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.976952 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-data\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977014 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.976998 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eedfea8-2d8c-4598-badd-d1063cc88c09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eedfea8-2d8c-4598-badd-d1063cc88c09-istiod-ca-cert\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977043 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-certs\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977043 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-socket" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:17.977071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977061 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-kube-api-access-4dqxh\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-kube-api-access-4dqxh\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977113 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-envoy\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977160 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-token\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-token\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977193 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-podinfo\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977234 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-credential-socket\") pod \"6eedfea8-2d8c-4598-badd-d1063cc88c09\" (UID: \"6eedfea8-2d8c-4598-badd-d1063cc88c09\") " Nov 03 21:44:17.977365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977235 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-data" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:17.977365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977292 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-certs" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:17.977557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977536 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-credential-socket" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:17.977799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977774 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eedfea8-2d8c-4598-badd-d1063cc88c09-istiod-ca-cert" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:17.977925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977901 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:17.978001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977931 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:17.978001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977947 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:17.978001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977961 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eedfea8-2d8c-4598-badd-d1063cc88c09-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:17.978001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.977975 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:17.979580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.979555 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-podinfo" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:44:17.979748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.979727 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-envoy" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:17.979881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.979864 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-token" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:17.980041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:17.980021 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-kube-api-access-4dqxh" pod "6eedfea8-2d8c-4598-badd-d1063cc88c09" (UID: "6eedfea8-2d8c-4598-badd-d1063cc88c09"). InnerVolumeSpecName "kube-api-access-4dqxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:18.078781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.078746 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-credential-socket\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.078956 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.078795 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-envoy\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.078956 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.078837 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-data\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.078956 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.078896 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istiod-ca-cert\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.078956 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.078944 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-kube-api-access-2g527\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-kube-api-access-2g527\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.079177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.078975 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-kube-api-access-thf8z\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-kube-api-access-thf8z\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.079177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079007 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-credential-socket\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.079177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079037 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-credential-socket" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.079177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079068 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-certs\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079377 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-data" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079552 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istiod-ca-cert" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079686 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-credential-socket" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079718 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-podinfo\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079766 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-data\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079797 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-envoy\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079873 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-certs\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079905 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-podinfo\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.079982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.079947 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-token\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-token\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.080534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.080004 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-token\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-token\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.080534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.080097 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-socket\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.080534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.080152 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-socket\") pod \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\" (UID: \"e8efb056-b5a5-4098-b8ea-dac1bc7f1522\") " Nov 03 21:44:18.080534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.080191 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istiod-ca-cert\") pod \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\" (UID: \"b9931e2f-5857-4964-b99b-fe8eb3b8a281\") " Nov 03 21:44:18.081079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081045 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-certs" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.081676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081378 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-envoy" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081506 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081853 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqxh\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-kube-api-access-4dqxh\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081877 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081904 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081920 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081933 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081947 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eedfea8-2d8c-4598-badd-d1063cc88c09-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081962 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081976 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081990 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081576 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-data" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.082011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081583 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-socket" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.082672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081705 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-socket" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.082672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081643 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-certs" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.082672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081855 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-kube-api-access-2g527" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "kube-api-access-2g527". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:18.082672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.081905 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istiod-ca-cert" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:18.083592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.083565 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-token" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:18.083808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.083780 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-token" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:18.083951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.083934 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-kube-api-access-thf8z" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "kube-api-access-thf8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:18.084005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.083944 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-podinfo" pod "b9931e2f-5857-4964-b99b-fe8eb3b8a281" (UID: "b9931e2f-5857-4964-b99b-fe8eb3b8a281"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:44:18.084903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.084886 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-podinfo" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:44:18.085575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.085554 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-envoy" pod "e8efb056-b5a5-4098-b8ea-dac1bc7f1522" (UID: "e8efb056-b5a5-4098-b8ea-dac1bc7f1522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:18.183370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183285 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183320 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183334 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183347 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183359 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183373 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183386 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9931e2f-5857-4964-b99b-fe8eb3b8a281-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183399 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183424 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9931e2f-5857-4964-b99b-fe8eb3b8a281-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183436 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g527\" (UniqueName: \"kubernetes.io/projected/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-kube-api-access-2g527\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183451 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thf8z\" (UniqueName: \"kubernetes.io/projected/b9931e2f-5857-4964-b99b-fe8eb3b8a281-kube-api-access-thf8z\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.183848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.183463 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8efb056-b5a5-4098-b8ea-dac1bc7f1522-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:18.809489 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.809412 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" Nov 03 21:44:18.809489 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.809427 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf" event={"ID":"e8efb056-b5a5-4098-b8ea-dac1bc7f1522","Type":"ContainerDied","Data":"d7c02856e2740f1c760a7398a6601d21f453c615e46da3aa621da97477bd83f7"} Nov 03 21:44:18.809973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.809502 5931 scope.go:122] "RemoveContainer" containerID="1132d54da47b7fe44c8e4ae442a4401fc9e4226e11f6ab74c68281de13956684" Nov 03 21:44:18.811132 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.811115 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" Nov 03 21:44:18.811247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.811129 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95" event={"ID":"b9931e2f-5857-4964-b99b-fe8eb3b8a281","Type":"ContainerDied","Data":"c0851356173772d148a49da0fff624c095c67bf3591e3987064e01dd9730100a"} Nov 03 21:44:18.813123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.813094 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" Nov 03 21:44:18.813237 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.813130 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc" event={"ID":"6eedfea8-2d8c-4598-badd-d1063cc88c09","Type":"ContainerDied","Data":"23df68d951f31daec61d9da3d48cbd33c751eaa135d88db8c5801dd0ab2079b6"} Nov 03 21:44:18.823776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.823696 5931 scope.go:122] "RemoveContainer" containerID="3526c912c77fdf6c2f041c19530199818951d55850aac26d61e84f3122db4557" Nov 03 21:44:18.832376 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.832346 5931 scope.go:122] "RemoveContainer" containerID="105a9c5ecdbc69a95b18261a418101fdfdf0f85706830f1800df687743d673db" Nov 03 21:44:18.972446 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:18.972421 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc"] Nov 03 21:44:19.006639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:19.006586 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:19.023317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:19.023289 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-mh9hc"] Nov 03 21:44:19.131385 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:19.131333 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf"] Nov 03 21:44:19.134738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:19.134712 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-wn2bf"] Nov 03 21:44:19.230091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:19.230066 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:19.242305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:19.242278 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95"] Nov 03 21:44:19.247778 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:19.247758 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-ppb95"] Nov 03 21:44:20.505439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:20.505406 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" path="/var/lib/kubelet/pods/6eedfea8-2d8c-4598-badd-d1063cc88c09/volumes" Nov 03 21:44:20.505867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:20.505851 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" path="/var/lib/kubelet/pods/b9931e2f-5857-4964-b99b-fe8eb3b8a281/volumes" Nov 03 21:44:20.506247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:20.506235 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" path="/var/lib/kubelet/pods/e8efb056-b5a5-4098-b8ea-dac1bc7f1522/volumes" Nov 03 21:44:21.851818 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:21.851758 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:22.280067 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.279983 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:22.505387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.505342 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.732832 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8"] Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733603 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733758 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733814 5931 cpu_manager.go:414] "containerMap: removing container" podUID="4ca62afe-8252-49db-99f9-d591f81b4392" containerName="payload-processing" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733822 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca62afe-8252-49db-99f9-d591f81b4392" containerName="payload-processing" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733837 5931 cpu_manager.go:414] "containerMap: removing container" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733846 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733858 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7b44aee0-da1d-4e4e-a775-0842abb2ae51" containerName="payload-pre-processing" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733865 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b44aee0-da1d-4e4e-a775-0842abb2ae51" containerName="payload-pre-processing" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733902 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" Nov 03 21:44:22.733991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.733909 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" Nov 03 21:44:22.734695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.734196 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9931e2f-5857-4964-b99b-fe8eb3b8a281" containerName="istio-proxy" Nov 03 21:44:22.734695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.734218 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eedfea8-2d8c-4598-badd-d1063cc88c09" containerName="istio-proxy" Nov 03 21:44:22.734695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.734236 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca62afe-8252-49db-99f9-d591f81b4392" containerName="payload-processing" Nov 03 21:44:22.734695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.734246 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8efb056-b5a5-4098-b8ea-dac1bc7f1522" containerName="istio-proxy" Nov 03 21:44:22.734695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.734255 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b44aee0-da1d-4e4e-a775-0842abb2ae51" containerName="payload-pre-processing" Nov 03 21:44:22.738215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.738153 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:22.770978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.770946 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8"] Nov 03 21:44:22.816864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.816840 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a1931b\"" Nov 03 21:44:22.817005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.816968 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a1931b-dockercfg-tztn2\"" Nov 03 21:44:22.832005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.831978 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7rpc\" (UniqueName: \"kubernetes.io/projected/822f3025-5d9e-460f-bf25-e2503a851e28-kube-api-access-g7rpc\") pod \"payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:22.832152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.832138 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822f3025-5d9e-460f-bf25-e2503a851e28-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:22.885325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.885298 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn"] Nov 03 21:44:22.890295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.890274 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:22.933696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.933670 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822f3025-5d9e-460f-bf25-e2503a851e28-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:22.933787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.933741 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7rpc\" (UniqueName: \"kubernetes.io/projected/822f3025-5d9e-460f-bf25-e2503a851e28-kube-api-access-g7rpc\") pod \"payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:22.934299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.934279 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822f3025-5d9e-460f-bf25-e2503a851e28-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:22.947258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.947236 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn"] Nov 03 21:44:22.990754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:22.990700 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7rpc\" (UniqueName: \"kubernetes.io/projected/822f3025-5d9e-460f-bf25-e2503a851e28-kube-api-access-g7rpc\") pod \"payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:23.034747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.034720 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183b89a1-6abe-49eb-923f-39c17a775ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.034843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.034768 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhndx\" (UniqueName: \"kubernetes.io/projected/183b89a1-6abe-49eb-923f-39c17a775ecb-kube-api-access-lhndx\") pod \"payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.058776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.058747 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:23.136030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.135999 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhndx\" (UniqueName: \"kubernetes.io/projected/183b89a1-6abe-49eb-923f-39c17a775ecb-kube-api-access-lhndx\") pod \"payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.136192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.136180 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183b89a1-6abe-49eb-923f-39c17a775ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.136947 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.136924 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183b89a1-6abe-49eb-923f-39c17a775ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.193229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.193199 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhndx\" (UniqueName: \"kubernetes.io/projected/183b89a1-6abe-49eb-923f-39c17a775ecb-kube-api-access-lhndx\") pod \"payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.204069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.204048 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.435459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.435431 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8"] Nov 03 21:44:23.437340 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:44:23.437315 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822f3025_5d9e_460f_bf25_e2503a851e28.slice/crio-8c0ab425686c0c0a77d8a8100dc1ba560ea92eea59829fb820d8992031cd0275 WatchSource:0}: Error finding container 8c0ab425686c0c0a77d8a8100dc1ba560ea92eea59829fb820d8992031cd0275: Status 404 returned error can't find the container with id 8c0ab425686c0c0a77d8a8100dc1ba560ea92eea59829fb820d8992031cd0275 Nov 03 21:44:23.577241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.577214 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn"] Nov 03 21:44:23.578902 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:44:23.578878 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183b89a1_6abe_49eb_923f_39c17a775ecb.slice/crio-e97be600f7184d45f92cc8efac6e054df991620ea1e349abcb4ddbdb58e4b191 WatchSource:0}: Error finding container e97be600f7184d45f92cc8efac6e054df991620ea1e349abcb4ddbdb58e4b191: Status 404 returned error can't find the container with id e97be600f7184d45f92cc8efac6e054df991620ea1e349abcb4ddbdb58e4b191 Nov 03 21:44:23.627321 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.627289 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:23.851469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.851435 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" event={"ID":"183b89a1-6abe-49eb-923f-39c17a775ecb","Type":"ContainerStarted","Data":"eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623"} Nov 03 21:44:23.851675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.851497 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" event={"ID":"183b89a1-6abe-49eb-923f-39c17a775ecb","Type":"ContainerStarted","Data":"e97be600f7184d45f92cc8efac6e054df991620ea1e349abcb4ddbdb58e4b191"} Nov 03 21:44:23.851675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.851635 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:23.853391 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.853367 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" event={"ID":"822f3025-5d9e-460f-bf25-e2503a851e28","Type":"ContainerStarted","Data":"51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27"} Nov 03 21:44:23.853538 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.853412 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" event={"ID":"822f3025-5d9e-460f-bf25-e2503a851e28","Type":"ContainerStarted","Data":"8c0ab425686c0c0a77d8a8100dc1ba560ea92eea59829fb820d8992031cd0275"} Nov 03 21:44:23.853538 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.853454 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:23.865037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.865011 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:23.989695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:23.989599 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" podStartSLOduration=1.989580066 podStartE2EDuration="1.989580066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:23.988295255 +0000 UTC m=+1731.599143071" watchObservedRunningTime="2026-11-03 21:44:23.989580066 +0000 UTC m=+1731.600427883" Nov 03 21:44:24.107314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:24.107224 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" podStartSLOduration=2.107205811 podStartE2EDuration="2.107205811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:24.104323869 +0000 UTC m=+1731.715171683" watchObservedRunningTime="2026-11-03 21:44:24.107205811 +0000 UTC m=+1731.718053758" Nov 03 21:44:24.127738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:24.127706 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:26.730654 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:26.730607 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:30.665113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:30.665084 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv"] Nov 03 21:44:30.665479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:30.665287 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" podUID="34277911-825b-4fb2-8fe2-2bea9eeadbc1" containerName="maas-api" containerID="cri-o://0b25ddff6836f111941ad7150c3894879cbf3bdf48d3b3d3501402683738f180" gracePeriod=30 Nov 03 21:44:30.864136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:30.861390 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:30.921585 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:30.921556 5931 generic.go:356] "Generic (PLEG): container finished" podID="34277911-825b-4fb2-8fe2-2bea9eeadbc1" containerID="0b25ddff6836f111941ad7150c3894879cbf3bdf48d3b3d3501402683738f180" exitCode=0 Nov 03 21:44:30.921730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:30.921602 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" event={"ID":"34277911-825b-4fb2-8fe2-2bea9eeadbc1","Type":"ContainerDied","Data":"0b25ddff6836f111941ad7150c3894879cbf3bdf48d3b3d3501402683738f180"} Nov 03 21:44:30.945564 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:30.945542 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:44:31.010512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.010475 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-metrics-certs\") pod \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " Nov 03 21:44:31.010721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.010547 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34277911-825b-4fb2-8fe2-2bea9eeadbc1-kube-api-access-gbzbx\" (UniqueName: \"kubernetes.io/projected/34277911-825b-4fb2-8fe2-2bea9eeadbc1-kube-api-access-gbzbx\") pod \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " Nov 03 21:44:31.010721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.010700 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls\") pod \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " Nov 03 21:44:31.010853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.010745 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34277911-825b-4fb2-8fe2-2bea9eeadbc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34277911-825b-4fb2-8fe2-2bea9eeadbc1-openshift-service-ca\") pod \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\" (UID: \"34277911-825b-4fb2-8fe2-2bea9eeadbc1\") " Nov 03 21:44:31.011180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.011152 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34277911-825b-4fb2-8fe2-2bea9eeadbc1-openshift-service-ca" pod "34277911-825b-4fb2-8fe2-2bea9eeadbc1" (UID: "34277911-825b-4fb2-8fe2-2bea9eeadbc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:31.011441 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.011420 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34277911-825b-4fb2-8fe2-2bea9eeadbc1-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:31.012895 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.012874 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34277911-825b-4fb2-8fe2-2bea9eeadbc1-kube-api-access-gbzbx" pod "34277911-825b-4fb2-8fe2-2bea9eeadbc1" (UID: "34277911-825b-4fb2-8fe2-2bea9eeadbc1"). InnerVolumeSpecName "kube-api-access-gbzbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:31.012978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.012889 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-metrics-certs" pod "34277911-825b-4fb2-8fe2-2bea9eeadbc1" (UID: "34277911-825b-4fb2-8fe2-2bea9eeadbc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:31.012978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.012906 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls" pod "34277911-825b-4fb2-8fe2-2bea9eeadbc1" (UID: "34277911-825b-4fb2-8fe2-2bea9eeadbc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:31.112571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.112541 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:31.112571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.112566 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzbx\" (UniqueName: \"kubernetes.io/projected/34277911-825b-4fb2-8fe2-2bea9eeadbc1-kube-api-access-gbzbx\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:31.112571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.112576 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34277911-825b-4fb2-8fe2-2bea9eeadbc1-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:31.928845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.928813 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" Nov 03 21:44:31.928845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.928830 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv" event={"ID":"34277911-825b-4fb2-8fe2-2bea9eeadbc1","Type":"ContainerDied","Data":"a35de3ca34f22ea7811b0eca50cd937634c0931ae7b8289761ad2aa9518caa33"} Nov 03 21:44:31.929342 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:31.928912 5931 scope.go:122] "RemoveContainer" containerID="0b25ddff6836f111941ad7150c3894879cbf3bdf48d3b3d3501402683738f180" Nov 03 21:44:32.093183 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:32.093144 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv"] Nov 03 21:44:32.140848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:32.140821 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad161-757d859954-r59jv"] Nov 03 21:44:32.506531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:32.506504 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34277911-825b-4fb2-8fe2-2bea9eeadbc1" path="/var/lib/kubelet/pods/34277911-825b-4fb2-8fe2-2bea9eeadbc1/volumes" Nov 03 21:44:32.665171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:32.665142 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:33.067593 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:44:33.067553 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:44:33.342004 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:33.341976 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:34.862331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:34.862304 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:44:34.862791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:34.862426 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:44:37.102147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:37.102118 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:38.743570 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:38.743537 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:38.982708 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:38.982676 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:39.248627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:39.248593 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:39.482109 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:39.482078 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:39.672166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:39.672128 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n"] Nov 03 21:44:39.672426 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:39.672397 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" containerID="cri-o://8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3" gracePeriod=30 Nov 03 21:44:39.708811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:39.708782 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:39.931473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:39.931387 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:41.963588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:41.963562 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:42.623766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:42.623727 5931 patch_prober.go:29] interesting pod/e2e-api-b-dad161-openshift-default-7476c569-xbg8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:44:42.623928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:42.623793 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:44:44.043702 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:44.043667 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:44.942182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:44.942124 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:44:45.016950 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.016926 5931 generic.go:356] "Generic (PLEG): container finished" podID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerID="8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3" exitCode=0 Nov 03 21:44:45.017066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.016976 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" Nov 03 21:44:45.017066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.017012 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" event={"ID":"754a057d-4185-4c25-aef2-8f2f0f0d335b","Type":"ContainerDied","Data":"8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3"} Nov 03 21:44:45.017144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.017088 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n" event={"ID":"754a057d-4185-4c25-aef2-8f2f0f0d335b","Type":"ContainerDied","Data":"cf2b67f2e6e8dc79d6f3d1c7ec41519ca41fefecb19fcbd047a4c0bdfac902b5"} Nov 03 21:44:45.017144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.017109 5931 scope.go:122] "RemoveContainer" containerID="8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3" Nov 03 21:44:45.026362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.026346 5931 scope.go:122] "RemoveContainer" containerID="c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb" Nov 03 21:44:45.035109 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.035093 5931 scope.go:122] "RemoveContainer" containerID="8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3" Nov 03 21:44:45.035397 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:44:45.035379 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3\": container with ID starting with 8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3 not found: ID does not exist" containerID="8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3" Nov 03 21:44:45.035445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.035405 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3"} err="failed to get container status \"8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3\": rpc error: code = NotFound desc = could not find container \"8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3\": container with ID starting with 8c8b902749143e5ae688e7129884dee7ee49a7e98554c7b6fd7094fac67eb2c3 not found: ID does not exist" Nov 03 21:44:45.035445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.035421 5931 scope.go:122] "RemoveContainer" containerID="c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb" Nov 03 21:44:45.035709 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:44:45.035682 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb\": container with ID starting with c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb not found: ID does not exist" containerID="c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb" Nov 03 21:44:45.035783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.035719 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb"} err="failed to get container status \"c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb\": rpc error: code = NotFound desc = could not find container \"c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb\": container with ID starting with c82821f3e1c3620179467aad6350520c9c8bba7872c7c7580650dc102684a4bb not found: ID does not exist" Nov 03 21:44:45.062120 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062099 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-envoy\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062129 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-socket\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062276 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-certs\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062312 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-kube-api-access-ph2g9\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-kube-api-access-ph2g9\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062353 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-podinfo\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062394 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-socket" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:45.062589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062414 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754a057d-4185-4c25-aef2-8f2f0f0d335b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754a057d-4185-4c25-aef2-8f2f0f0d335b-istiod-ca-cert\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062470 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-token\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-token\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062501 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-data\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.062589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062551 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-certs" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:45.062816 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.062651 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-credential-socket\") pod \"754a057d-4185-4c25-aef2-8f2f0f0d335b\" (UID: \"754a057d-4185-4c25-aef2-8f2f0f0d335b\") " Nov 03 21:44:45.063017 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.063003 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.063056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.063021 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.063252 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.063220 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-data" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:45.063363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.063347 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-credential-socket" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:45.063418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.063374 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754a057d-4185-4c25-aef2-8f2f0f0d335b-istiod-ca-cert" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:45.064520 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.064485 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-envoy" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:44:45.065076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.065052 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-kube-api-access-ph2g9" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "kube-api-access-ph2g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:45.065210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.065191 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-token" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:45.065273 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.065255 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-podinfo" pod "754a057d-4185-4c25-aef2-8f2f0f0d335b" (UID: "754a057d-4185-4c25-aef2-8f2f0f0d335b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:44:45.163945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.163919 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph2g9\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-kube-api-access-ph2g9\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.163945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.163945 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.164106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.163955 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754a057d-4185-4c25-aef2-8f2f0f0d335b-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.164106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.163963 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.164106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.163972 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.164106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.163980 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.164106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.163988 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754a057d-4185-4c25-aef2-8f2f0f0d335b-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:44:45.387381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.387324 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:45.440271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.440237 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n"] Nov 03 21:44:45.443525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.443501 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dad161-openshift-default-7476c569-xbg8n"] Nov 03 21:44:45.624899 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.624875 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:45.850007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:45.849974 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:46.505696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:46.505664 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" path="/var/lib/kubelet/pods/754a057d-4185-4c25-aef2-8f2f0f0d335b/volumes" Nov 03 21:44:49.023364 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:49.023329 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:54.128970 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:54.128934 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:55.475187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.475153 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:55.907966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.907932 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm"] Nov 03 21:44:55.908485 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.908467 5931 cpu_manager.go:414] "containerMap: removing container" podUID="34277911-825b-4fb2-8fe2-2bea9eeadbc1" containerName="maas-api" Nov 03 21:44:55.908583 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.908489 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34277911-825b-4fb2-8fe2-2bea9eeadbc1" containerName="maas-api" Nov 03 21:44:55.908583 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.908514 5931 cpu_manager.go:414] "containerMap: removing container" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" Nov 03 21:44:55.908583 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.908521 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" Nov 03 21:44:55.908793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.908740 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" Nov 03 21:44:55.908793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.908758 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="34277911-825b-4fb2-8fe2-2bea9eeadbc1" containerName="maas-api" Nov 03 21:44:55.912182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.912165 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.993160 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.993123 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.993495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.993471 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bl7\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-kube-api-access-n5bl7\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.993695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.993676 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.994133 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.994111 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.994303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.994284 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.994382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.994326 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.994382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.994357 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.994572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.994387 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972de09d-80a9-4a4c-b226-53933aae736f-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:55.994572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:55.994516 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972de09d-80a9-4a4c-b226-53933aae736f-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.026676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.026640 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm"] Nov 03 21:44:56.035470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.035436 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-87e1cf-openshift-default-dockercfg-28lnl\"" Nov 03 21:44:56.095416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.095377 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972de09d-80a9-4a4c-b226-53933aae736f-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.095631 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.095472 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.095631 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.095512 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bl7\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-kube-api-access-n5bl7\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.095631 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.095574 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.095949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.095927 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.095928 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096005 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096122 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096201 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096164 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096198 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096232 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972de09d-80a9-4a4c-b226-53933aae736f-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096249 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096585 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096564 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.096870 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.096851 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972de09d-80a9-4a4c-b226-53933aae736f-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.098144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.098123 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972de09d-80a9-4a4c-b226-53933aae736f-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.098318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.098301 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:56.098658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.098640 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.152998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.152963 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.153152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.153085 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bl7\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-kube-api-access-n5bl7\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.226196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.226105 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:56.331584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.331554 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:56.656003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.655982 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm"] Nov 03 21:44:56.658124 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:44:56.658099 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972de09d_80a9_4a4c_b226_53933aae736f.slice/crio-c587f0a05fd6be3e365730449bd1011414bf331c55907b85a965a6ea0697c11b WatchSource:0}: Error finding container c587f0a05fd6be3e365730449bd1011414bf331c55907b85a965a6ea0697c11b: Status 404 returned error can't find the container with id c587f0a05fd6be3e365730449bd1011414bf331c55907b85a965a6ea0697c11b Nov 03 21:44:56.660190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.660162 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:44:56.660263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.660224 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:44:56.660263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:56.660250 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:44:57.089796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:57.089753 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" event={"ID":"972de09d-80a9-4a4c-b226-53933aae736f","Type":"ContainerStarted","Data":"0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b"} Nov 03 21:44:57.089975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:57.089823 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" event={"ID":"972de09d-80a9-4a4c-b226-53933aae736f","Type":"ContainerStarted","Data":"c587f0a05fd6be3e365730449bd1011414bf331c55907b85a965a6ea0697c11b"} Nov 03 21:44:57.207749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:57.207697 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podStartSLOduration=2.207683102 podStartE2EDuration="2.207683102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:57.20429647 +0000 UTC m=+1764.815144285" watchObservedRunningTime="2026-11-03 21:44:57.207683102 +0000 UTC m=+1764.818530917" Nov 03 21:44:57.226598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:57.226573 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:57.232038 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:57.232019 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:57.757006 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:57.756967 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:57.996287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:57.995825 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:58.097539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.097495 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:58.098649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.098601 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:44:58.746765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.746733 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg"] Nov 03 21:44:58.747421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.747405 5931 cpu_manager.go:414] "containerMap: removing container" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" Nov 03 21:44:58.747473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.747424 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" Nov 03 21:44:58.747531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.747523 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="754a057d-4185-4c25-aef2-8f2f0f0d335b" containerName="istio-proxy" Nov 03 21:44:58.750968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.750952 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.833899 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.833861 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twct8\" (UniqueName: \"kubernetes.io/projected/7a9552d8-1a8c-431a-8289-814223f6ec2c-kube-api-access-twct8\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.834344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.834016 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-metrics-certs\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.834344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.834065 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9552d8-1a8c-431a-8289-814223f6ec2c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.834344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.834148 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.853339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.853312 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a1931b-serving-cert\"" Nov 03 21:44:58.856802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.856777 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg"] Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.935222 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-metrics-certs\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.935283 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9552d8-1a8c-431a-8289-814223f6ec2c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.935364 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.935493 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twct8\" (UniqueName: \"kubernetes.io/projected/7a9552d8-1a8c-431a-8289-814223f6ec2c-kube-api-access-twct8\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.937006 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9552d8-1a8c-431a-8289-814223f6ec2c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:44:58.937107 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-serving-cert: secret "maas-api-e2e-api-a-a1931b-serving-cert" not found Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:44:58.937161 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls podName:7a9552d8-1a8c-431a-8289-814223f6ec2c nodeName:}" failed. No retries permitted until 2026-11-03 21:44:59.437142933 +0000 UTC m=+1767.047990731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls") pod "maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" (UID: "7a9552d8-1a8c-431a-8289-814223f6ec2c") : secret "maas-api-e2e-api-a-a1931b-serving-cert" not found Nov 03 21:44:58.950023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.944410 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-metrics-certs\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:58.992598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:58.992558 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twct8\" (UniqueName: \"kubernetes.io/projected/7a9552d8-1a8c-431a-8289-814223f6ec2c-kube-api-access-twct8\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:59.036051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:59.035956 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:59.270350 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:59.270297 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:59.442455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:59.442420 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:59.461251 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:59.461222 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls\") pod \"maas-api-e2e-api-a-a1931b-7dc784d965-q8njg\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:44:59.499726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:59.499695 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:59.664637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:44:59.664574 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:45:00.080196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.080164 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg"] Nov 03 21:45:00.083197 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:00.083152 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9552d8_1a8c_431a_8289_814223f6ec2c.slice/crio-db01795efedd4f4225ff960f97f7153d1a3cfce14456cf9ff5845a7945bda2f8 WatchSource:0}: Error finding container db01795efedd4f4225ff960f97f7153d1a3cfce14456cf9ff5845a7945bda2f8: Status 404 returned error can't find the container with id db01795efedd4f4225ff960f97f7153d1a3cfce14456cf9ff5845a7945bda2f8 Nov 03 21:45:00.111062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.111038 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" event={"ID":"7a9552d8-1a8c-431a-8289-814223f6ec2c","Type":"ContainerStarted","Data":"db01795efedd4f4225ff960f97f7153d1a3cfce14456cf9ff5845a7945bda2f8"} Nov 03 21:45:00.176989 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.176954 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps"] Nov 03 21:45:00.182313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.182290 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f"] Nov 03 21:45:00.182479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.182418 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.186500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.186475 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.234341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.234296 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\"" Nov 03 21:45:00.262564 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.262516 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d746f8a-820c-4509-ab47-cb681d12c47e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.262564 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.262561 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.262837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.262769 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrzh\" (UniqueName: \"kubernetes.io/projected/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-kube-api-access-whrzh\") pod \"maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.262995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.262975 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcn7d\" (UniqueName: \"kubernetes.io/projected/2d746f8a-820c-4509-ab47-cb681d12c47e-kube-api-access-mcn7d\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.314675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.314642 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f"] Nov 03 21:45:00.364378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.364348 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcn7d\" (UniqueName: \"kubernetes.io/projected/2d746f8a-820c-4509-ab47-cb681d12c47e-kube-api-access-mcn7d\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.364562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.364437 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d746f8a-820c-4509-ab47-cb681d12c47e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.364562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.364460 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.364562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.364514 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrzh\" (UniqueName: \"kubernetes.io/projected/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-kube-api-access-whrzh\") pod \"maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.365294 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.365270 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d746f8a-820c-4509-ab47-cb681d12c47e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.365404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.365312 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.382269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.382244 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps"] Nov 03 21:45:00.421080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.421024 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrzh\" (UniqueName: \"kubernetes.io/projected/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-kube-api-access-whrzh\") pod \"maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.421228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.421139 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcn7d\" (UniqueName: \"kubernetes.io/projected/2d746f8a-820c-4509-ab47-cb681d12c47e-kube-api-access-mcn7d\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.499373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.499338 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:00.506507 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.506483 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:00.583332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.583301 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:00.836459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.836367 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:00.883789 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.883752 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps"] Nov 03 21:45:00.886391 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:00.886365 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d746f8a_820c_4509_ab47_cb681d12c47e.slice/crio-8a17e5c266b0305bc5055112b7f314b5d6d27b10ce7d4af00b39c7694ccd3a23 WatchSource:0}: Error finding container 8a17e5c266b0305bc5055112b7f314b5d6d27b10ce7d4af00b39c7694ccd3a23: Status 404 returned error can't find the container with id 8a17e5c266b0305bc5055112b7f314b5d6d27b10ce7d4af00b39c7694ccd3a23 Nov 03 21:45:00.933169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:00.933144 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f"] Nov 03 21:45:00.935945 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:00.935914 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf8a456_7bf5_4ac9_90f7_3e5ed717c857.slice/crio-05db9ad0382295873aaf58e6aad6fd824c39a41f3fba77b6b561ffaf5b4ddbca WatchSource:0}: Error finding container 05db9ad0382295873aaf58e6aad6fd824c39a41f3fba77b6b561ffaf5b4ddbca: Status 404 returned error can't find the container with id 05db9ad0382295873aaf58e6aad6fd824c39a41f3fba77b6b561ffaf5b4ddbca Nov 03 21:45:01.119465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:01.119392 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" event={"ID":"2d746f8a-820c-4509-ab47-cb681d12c47e","Type":"ContainerStarted","Data":"8a17e5c266b0305bc5055112b7f314b5d6d27b10ce7d4af00b39c7694ccd3a23"} Nov 03 21:45:01.121377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:01.121349 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" event={"ID":"7a9552d8-1a8c-431a-8289-814223f6ec2c","Type":"ContainerStarted","Data":"ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef"} Nov 03 21:45:01.121536 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:01.121408 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:45:01.122812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:01.122786 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" event={"ID":"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857","Type":"ContainerStarted","Data":"05db9ad0382295873aaf58e6aad6fd824c39a41f3fba77b6b561ffaf5b4ddbca"} Nov 03 21:45:01.237472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:01.237395 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" podStartSLOduration=2.338988293 podStartE2EDuration="3.237375197s" totalImagesPullingTime="898.386904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.235603982 +0000 UTC m=+1768.846451827" watchObservedRunningTime="2026-11-03 21:45:01.237375197 +0000 UTC m=+1768.848223017" Nov 03 21:45:02.135220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:02.132177 5931 generic.go:356] "Generic (PLEG): container finished" podID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerID="d70df37a8d99663ec82887b973afda8ee779764bde0c814c817090f0b533d78f" exitCode=60 Nov 03 21:45:02.135220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:02.132309 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" event={"ID":"2d746f8a-820c-4509-ab47-cb681d12c47e","Type":"ContainerDied","Data":"d70df37a8d99663ec82887b973afda8ee779764bde0c814c817090f0b533d78f"} Nov 03 21:45:02.135220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:02.132807 5931 scope.go:122] "RemoveContainer" containerID="d70df37a8d99663ec82887b973afda8ee779764bde0c814c817090f0b533d78f" Nov 03 21:45:02.142865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:02.138155 5931 generic.go:356] "Generic (PLEG): container finished" podID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerID="eef106ffd8b03a2a916fee0d3cca5133701195982eee46a23ac7da6d762e2906" exitCode=60 Nov 03 21:45:02.142865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:02.139047 5931 scope.go:122] "RemoveContainer" containerID="eef106ffd8b03a2a916fee0d3cca5133701195982eee46a23ac7da6d762e2906" Nov 03 21:45:02.142865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:02.139304 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" event={"ID":"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857","Type":"ContainerDied","Data":"eef106ffd8b03a2a916fee0d3cca5133701195982eee46a23ac7da6d762e2906"} Nov 03 21:45:03.070704 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:03.070660 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:45:03.147020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:03.146980 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" event={"ID":"2d746f8a-820c-4509-ab47-cb681d12c47e","Type":"ContainerStarted","Data":"9a0c750703691b8bd90261d81f9a35e8b0a072931c785ace7561842e544fe9df"} Nov 03 21:45:03.149130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:03.149102 5931 generic.go:356] "Generic (PLEG): container finished" podID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerID="e83dc172fd2cd64e81ff2238ae6ff6596018042702eebee7501225d21bf5c201" exitCode=60 Nov 03 21:45:03.149248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:03.149181 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" event={"ID":"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857","Type":"ContainerDied","Data":"e83dc172fd2cd64e81ff2238ae6ff6596018042702eebee7501225d21bf5c201"} Nov 03 21:45:03.149248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:03.149239 5931 scope.go:122] "RemoveContainer" containerID="eef106ffd8b03a2a916fee0d3cca5133701195982eee46a23ac7da6d762e2906" Nov 03 21:45:03.149515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:03.149488 5931 scope.go:122] "RemoveContainer" containerID="e83dc172fd2cd64e81ff2238ae6ff6596018042702eebee7501225d21bf5c201" Nov 03 21:45:03.149875 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:03.149847 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f_odh-ai-gateway-infra(6cf8a456-7bf5-4ac9-90f7-3e5ed717c857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" Nov 03 21:45:03.260496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:03.260445 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" podStartSLOduration=3.260432937 podStartE2EDuration="3.260432937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:03.258203703 +0000 UTC m=+1770.869051517" watchObservedRunningTime="2026-11-03 21:45:03.260432937 +0000 UTC m=+1770.871280784" Nov 03 21:45:03.971409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:03.971376 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:04.182554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.182129 5931 generic.go:356] "Generic (PLEG): container finished" podID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerID="9a0c750703691b8bd90261d81f9a35e8b0a072931c785ace7561842e544fe9df" exitCode=60 Nov 03 21:45:04.183027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.182918 5931 scope.go:122] "RemoveContainer" containerID="9a0c750703691b8bd90261d81f9a35e8b0a072931c785ace7561842e544fe9df" Nov 03 21:45:04.183328 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:04.183296 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps_odh-ai-gateway-infra(2d746f8a-820c-4509-ab47-cb681d12c47e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" Nov 03 21:45:04.183554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.183530 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" event={"ID":"2d746f8a-820c-4509-ab47-cb681d12c47e","Type":"ContainerDied","Data":"9a0c750703691b8bd90261d81f9a35e8b0a072931c785ace7561842e544fe9df"} Nov 03 21:45:04.183668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.183596 5931 scope.go:122] "RemoveContainer" containerID="d70df37a8d99663ec82887b973afda8ee779764bde0c814c817090f0b533d78f" Nov 03 21:45:04.187676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.187601 5931 scope.go:122] "RemoveContainer" containerID="e83dc172fd2cd64e81ff2238ae6ff6596018042702eebee7501225d21bf5c201" Nov 03 21:45:04.188005 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:04.187978 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f_odh-ai-gateway-infra(6cf8a456-7bf5-4ac9-90f7-3e5ed717c857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" Nov 03 21:45:04.270215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.270137 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47"] Nov 03 21:45:04.271396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.271365 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:04.275271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.275253 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:04.342918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.342875 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-87e1cf\"" Nov 03 21:45:04.344798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.344545 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47"] Nov 03 21:45:04.429589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.429521 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj"] Nov 03 21:45:04.442719 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.442685 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.457475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.455762 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f4508e-ba50-469e-b9c6-60c528f613bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:04.457475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.455948 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7ll\" (UniqueName: \"kubernetes.io/projected/72f4508e-ba50-469e-b9c6-60c528f613bc-kube-api-access-bs7ll\") pod \"payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:04.492136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.492104 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj"] Nov 03 21:45:04.495289 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.495262 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-87e1cf-dockercfg-jg4lf\"" Nov 03 21:45:04.573681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.569578 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7ll\" (UniqueName: \"kubernetes.io/projected/72f4508e-ba50-469e-b9c6-60c528f613bc-kube-api-access-bs7ll\") pod \"payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:04.573681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.569701 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9dsn\" (UniqueName: \"kubernetes.io/projected/59960395-b38c-4814-8fa6-8918fb2193f6-kube-api-access-n9dsn\") pod \"payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.573681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.569803 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f4508e-ba50-469e-b9c6-60c528f613bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:04.573681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.569831 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59960395-b38c-4814-8fa6-8918fb2193f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.573681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.571054 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f4508e-ba50-469e-b9c6-60c528f613bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:04.647633 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.646280 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7ll\" (UniqueName: \"kubernetes.io/projected/72f4508e-ba50-469e-b9c6-60c528f613bc-kube-api-access-bs7ll\") pod \"payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:04.704637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.691746 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9dsn\" (UniqueName: \"kubernetes.io/projected/59960395-b38c-4814-8fa6-8918fb2193f6-kube-api-access-n9dsn\") pod \"payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.705034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.705008 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59960395-b38c-4814-8fa6-8918fb2193f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.706250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.706131 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59960395-b38c-4814-8fa6-8918fb2193f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.766641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.766544 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9dsn\" (UniqueName: \"kubernetes.io/projected/59960395-b38c-4814-8fa6-8918fb2193f6-kube-api-access-n9dsn\") pod \"payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.769901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.769866 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:04.854312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.854189 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:04.895656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:04.891803 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:05.195768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:05.195737 5931 scope.go:122] "RemoveContainer" containerID="9a0c750703691b8bd90261d81f9a35e8b0a072931c785ace7561842e544fe9df" Nov 03 21:45:05.196673 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:05.196601 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps_odh-ai-gateway-infra(2d746f8a-820c-4509-ab47-cb681d12c47e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" Nov 03 21:45:05.279256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:05.279232 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj"] Nov 03 21:45:05.280976 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:05.280951 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59960395_b38c_4814_8fa6_8918fb2193f6.slice/crio-b1150588eed89de39f03b011858a9d88af3ad1fcd07f14008ff47ba637e38c6f WatchSource:0}: Error finding container b1150588eed89de39f03b011858a9d88af3ad1fcd07f14008ff47ba637e38c6f: Status 404 returned error can't find the container with id b1150588eed89de39f03b011858a9d88af3ad1fcd07f14008ff47ba637e38c6f Nov 03 21:45:05.410353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:05.410323 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47"] Nov 03 21:45:05.413294 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:05.413256 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f4508e_ba50_469e_b9c6_60c528f613bc.slice/crio-e4adea271269c1aa00e0812de451a452d2074a504b2286b591b0251c930a4ff2 WatchSource:0}: Error finding container e4adea271269c1aa00e0812de451a452d2074a504b2286b591b0251c930a4ff2: Status 404 returned error can't find the container with id e4adea271269c1aa00e0812de451a452d2074a504b2286b591b0251c930a4ff2 Nov 03 21:45:06.203137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.203098 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" event={"ID":"72f4508e-ba50-469e-b9c6-60c528f613bc","Type":"ContainerStarted","Data":"d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df"} Nov 03 21:45:06.203577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.203151 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" event={"ID":"72f4508e-ba50-469e-b9c6-60c528f613bc","Type":"ContainerStarted","Data":"e4adea271269c1aa00e0812de451a452d2074a504b2286b591b0251c930a4ff2"} Nov 03 21:45:06.203577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.203288 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:06.204759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.204735 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" event={"ID":"59960395-b38c-4814-8fa6-8918fb2193f6","Type":"ContainerStarted","Data":"930c314af8a1821fce13e950cbcb9cd4a25e081a65b7705817e2165e7aee0951"} Nov 03 21:45:06.204829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.204780 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" event={"ID":"59960395-b38c-4814-8fa6-8918fb2193f6","Type":"ContainerStarted","Data":"b1150588eed89de39f03b011858a9d88af3ad1fcd07f14008ff47ba637e38c6f"} Nov 03 21:45:06.204829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.204808 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:06.316736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.316694 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" podStartSLOduration=2.316681434 podStartE2EDuration="2.316681434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:06.316201711 +0000 UTC m=+1773.927049528" watchObservedRunningTime="2026-11-03 21:45:06.316681434 +0000 UTC m=+1773.927529247" Nov 03 21:45:06.433021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:06.432959 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" podStartSLOduration=2.432942082 podStartE2EDuration="2.432942082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:06.428216621 +0000 UTC m=+1774.039064457" watchObservedRunningTime="2026-11-03 21:45:06.432942082 +0000 UTC m=+1774.043789898" Nov 03 21:45:07.143723 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:07.143692 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:45:10.068168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:10.068135 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:15.234502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:15.234472 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:15.498885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:15.498807 5931 scope.go:122] "RemoveContainer" containerID="9a0c750703691b8bd90261d81f9a35e8b0a072931c785ace7561842e544fe9df" Nov 03 21:45:16.265768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:16.265738 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" event={"ID":"2d746f8a-820c-4509-ab47-cb681d12c47e","Type":"ContainerStarted","Data":"5a101d65316dc9aa7bcd6078bea9276f3e954b0eaa80714980004eac68e8af7a"} Nov 03 21:45:17.213580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.213553 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 21:45:17.213805 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.213663 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 21:45:17.273238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.273211 5931 generic.go:356] "Generic (PLEG): container finished" podID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerID="5a101d65316dc9aa7bcd6078bea9276f3e954b0eaa80714980004eac68e8af7a" exitCode=60 Nov 03 21:45:17.273664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.273283 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" event={"ID":"2d746f8a-820c-4509-ab47-cb681d12c47e","Type":"ContainerDied","Data":"5a101d65316dc9aa7bcd6078bea9276f3e954b0eaa80714980004eac68e8af7a"} Nov 03 21:45:17.273664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.273336 5931 scope.go:122] "RemoveContainer" containerID="9a0c750703691b8bd90261d81f9a35e8b0a072931c785ace7561842e544fe9df" Nov 03 21:45:17.273664 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.273650 5931 scope.go:122] "RemoveContainer" containerID="5a101d65316dc9aa7bcd6078bea9276f3e954b0eaa80714980004eac68e8af7a" Nov 03 21:45:17.273900 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:17.273880 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps_odh-ai-gateway-infra(2d746f8a-820c-4509-ab47-cb681d12c47e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" Nov 03 21:45:17.388908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.388878 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps"] Nov 03 21:45:17.498808 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:17.498741 5931 scope.go:122] "RemoveContainer" containerID="e83dc172fd2cd64e81ff2238ae6ff6596018042702eebee7501225d21bf5c201" Nov 03 21:45:18.282783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.282748 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" event={"ID":"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857","Type":"ContainerStarted","Data":"7cfd976458f1a0f400384c6148ae9a77bd699df596195c6b141e4b26621a71f3"} Nov 03 21:45:18.416734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.416712 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:18.505380 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.505331 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" podStartSLOduration=18.505318532 podStartE2EDuration="18.505318532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:18.503370789 +0000 UTC m=+1786.114218604" watchObservedRunningTime="2026-11-03 21:45:18.505318532 +0000 UTC m=+1786.116166383" Nov 03 21:45:18.567044 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.566992 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d746f8a-820c-4509-ab47-cb681d12c47e-kube-api-access-mcn7d\" (UniqueName: \"kubernetes.io/projected/2d746f8a-820c-4509-ab47-cb681d12c47e-kube-api-access-mcn7d\") pod \"2d746f8a-820c-4509-ab47-cb681d12c47e\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " Nov 03 21:45:18.567167 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.567087 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d746f8a-820c-4509-ab47-cb681d12c47e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d746f8a-820c-4509-ab47-cb681d12c47e-maas-api-service-ca\") pod \"2d746f8a-820c-4509-ab47-cb681d12c47e\" (UID: \"2d746f8a-820c-4509-ab47-cb681d12c47e\") " Nov 03 21:45:18.567428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.567407 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d746f8a-820c-4509-ab47-cb681d12c47e-maas-api-service-ca" pod "2d746f8a-820c-4509-ab47-cb681d12c47e" (UID: "2d746f8a-820c-4509-ab47-cb681d12c47e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:18.567691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.567668 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d746f8a-820c-4509-ab47-cb681d12c47e-maas-api-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:45:18.569229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.569206 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d746f8a-820c-4509-ab47-cb681d12c47e-kube-api-access-mcn7d" pod "2d746f8a-820c-4509-ab47-cb681d12c47e" (UID: "2d746f8a-820c-4509-ab47-cb681d12c47e"). InnerVolumeSpecName "kube-api-access-mcn7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:18.668382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:18.668355 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcn7d\" (UniqueName: \"kubernetes.io/projected/2d746f8a-820c-4509-ab47-cb681d12c47e-kube-api-access-mcn7d\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:45:19.290840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.290813 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" Nov 03 21:45:19.290840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.290831 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps" event={"ID":"2d746f8a-820c-4509-ab47-cb681d12c47e","Type":"ContainerDied","Data":"8a17e5c266b0305bc5055112b7f314b5d6d27b10ce7d4af00b39c7694ccd3a23"} Nov 03 21:45:19.291313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.290893 5931 scope.go:122] "RemoveContainer" containerID="5a101d65316dc9aa7bcd6078bea9276f3e954b0eaa80714980004eac68e8af7a" Nov 03 21:45:19.293154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.293134 5931 generic.go:356] "Generic (PLEG): container finished" podID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerID="7cfd976458f1a0f400384c6148ae9a77bd699df596195c6b141e4b26621a71f3" exitCode=60 Nov 03 21:45:19.293271 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.293198 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" event={"ID":"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857","Type":"ContainerDied","Data":"7cfd976458f1a0f400384c6148ae9a77bd699df596195c6b141e4b26621a71f3"} Nov 03 21:45:19.293671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.293654 5931 scope.go:122] "RemoveContainer" containerID="7cfd976458f1a0f400384c6148ae9a77bd699df596195c6b141e4b26621a71f3" Nov 03 21:45:19.293974 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:19.293954 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f_odh-ai-gateway-infra(6cf8a456-7bf5-4ac9-90f7-3e5ed717c857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" Nov 03 21:45:19.300968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.300952 5931 scope.go:122] "RemoveContainer" containerID="e83dc172fd2cd64e81ff2238ae6ff6596018042702eebee7501225d21bf5c201" Nov 03 21:45:19.370197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.370175 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:19.459488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.459430 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps"] Nov 03 21:45:19.462313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.462288 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895705-m2zps"] Nov 03 21:45:19.513429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.513402 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f"] Nov 03 21:45:19.580371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.580268 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8"] Nov 03 21:45:19.581208 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581181 5931 cpu_manager.go:414] "containerMap: removing container" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581208 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581208 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581226 5931 cpu_manager.go:414] "containerMap: removing container" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581234 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581258 5931 cpu_manager.go:414] "containerMap: removing container" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581265 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581396 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581409 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.581678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.581420 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" containerName="cleanup" Nov 03 21:45:19.586790 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.586767 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.596693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.596669 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm"] Nov 03 21:45:19.601139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.601118 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.645752 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.645722 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8"] Nov 03 21:45:19.682134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682095 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.682322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682143 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.682322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682242 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.682322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682288 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.682482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682379 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.682482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682469 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.682603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682511 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.682603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682566 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.682603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682588 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.682782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682676 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.682782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682705 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b7afa-0ed9-4294-b034-a178742f9539-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.682882 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682863 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxkz\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-kube-api-access-dxxkz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.682936 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682898 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpghw\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-kube-api-access-dpghw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.682936 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682928 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.683039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682965 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.683039 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.682988 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.683139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.683073 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b7afa-0ed9-4294-b034-a178742f9539-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.683139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.683133 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.713640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.713342 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm"] Nov 03 21:45:19.784734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.784703 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxkz\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-kube-api-access-dxxkz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.784734 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.784739 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpghw\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-kube-api-access-dpghw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.784988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.784759 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.784988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.784784 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.784988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.784941 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.784998 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b7afa-0ed9-4294-b034-a178742f9539-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785044 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.785164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785074 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785090 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785125 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785148 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785170 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785204 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785244 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785286 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785324 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785326 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785351 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785366 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785385 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785428 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b7afa-0ed9-4294-b034-a178742f9539-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785428 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.785528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785435 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.786223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.785707 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.786223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.786046 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.786380 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.786248 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.786380 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.786365 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b7afa-0ed9-4294-b034-a178742f9539-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.786666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.786643 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.787571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.787547 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.787765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.787749 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.787828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.787796 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b7afa-0ed9-4294-b034-a178742f9539-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.788221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.788205 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.841554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.841529 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpghw\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-kube-api-access-dpghw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.842557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.842529 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxkz\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-kube-api-access-dxxkz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.843096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.843076 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-b89cm\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:19.843492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.843469 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-bm7r8\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.905641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.905582 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:19.917830 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:19.917812 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:20.351545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.351518 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8"] Nov 03 21:45:20.354709 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:20.354685 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618b7afa_0ed9_4294_b034_a178742f9539.slice/crio-eb2b3cec6ed2d617bcd6fa0088eb3b50bd53f46536dcc5b6acc07f22acd870a0 WatchSource:0}: Error finding container eb2b3cec6ed2d617bcd6fa0088eb3b50bd53f46536dcc5b6acc07f22acd870a0: Status 404 returned error can't find the container with id eb2b3cec6ed2d617bcd6fa0088eb3b50bd53f46536dcc5b6acc07f22acd870a0 Nov 03 21:45:20.356394 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:20.356373 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cc75cb_312c_46d9_9926_72ccfd720f3c.slice/crio-9480e271cd133fd1095f014caf72a9da6ad4dea7d4d80048191fa24523af989a WatchSource:0}: Error finding container 9480e271cd133fd1095f014caf72a9da6ad4dea7d4d80048191fa24523af989a: Status 404 returned error can't find the container with id 9480e271cd133fd1095f014caf72a9da6ad4dea7d4d80048191fa24523af989a Nov 03 21:45:20.357745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.357711 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:20.357845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.357791 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:20.357845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.357835 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:20.359029 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.358996 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:20.359135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.359078 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:20.359135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.359122 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:20.400626 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.400579 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm"] Nov 03 21:45:20.444386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.444364 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:20.508126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.508001 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d746f8a-820c-4509-ab47-cb681d12c47e" path="/var/lib/kubelet/pods/2d746f8a-820c-4509-ab47-cb681d12c47e/volumes" Nov 03 21:45:20.597819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.597790 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-maas-api-service-ca\") pod \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " Nov 03 21:45:20.598000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.597977 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-kube-api-access-whrzh\" (UniqueName: \"kubernetes.io/projected/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-kube-api-access-whrzh\") pod \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\" (UID: \"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857\") " Nov 03 21:45:20.598482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.598456 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-maas-api-service-ca" pod "6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" (UID: "6cf8a456-7bf5-4ac9-90f7-3e5ed717c857"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:20.598592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.598554 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-maas-api-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:45:20.601044 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.601013 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-kube-api-access-whrzh" pod "6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" (UID: "6cf8a456-7bf5-4ac9-90f7-3e5ed717c857"). InnerVolumeSpecName "kube-api-access-whrzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:20.699989 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.699954 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrzh\" (UniqueName: \"kubernetes.io/projected/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857-kube-api-access-whrzh\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:45:20.929694 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:20.927106 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:21.168640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.167877 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:21.321418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.321312 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" event={"ID":"f9cc75cb-312c-46d9-9926-72ccfd720f3c","Type":"ContainerStarted","Data":"8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b"} Nov 03 21:45:21.321418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.321407 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" event={"ID":"f9cc75cb-312c-46d9-9926-72ccfd720f3c","Type":"ContainerStarted","Data":"9480e271cd133fd1095f014caf72a9da6ad4dea7d4d80048191fa24523af989a"} Nov 03 21:45:21.359642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.358861 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" event={"ID":"618b7afa-0ed9-4294-b034-a178742f9539","Type":"ContainerStarted","Data":"af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41"} Nov 03 21:45:21.359642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.358935 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" event={"ID":"618b7afa-0ed9-4294-b034-a178742f9539","Type":"ContainerStarted","Data":"eb2b3cec6ed2d617bcd6fa0088eb3b50bd53f46536dcc5b6acc07f22acd870a0"} Nov 03 21:45:21.364841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.362781 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" event={"ID":"6cf8a456-7bf5-4ac9-90f7-3e5ed717c857","Type":"ContainerDied","Data":"05db9ad0382295873aaf58e6aad6fd824c39a41f3fba77b6b561ffaf5b4ddbca"} Nov 03 21:45:21.364841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.362844 5931 scope.go:122] "RemoveContainer" containerID="7cfd976458f1a0f400384c6148ae9a77bd699df596195c6b141e4b26621a71f3" Nov 03 21:45:21.364841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.363004 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f" Nov 03 21:45:21.546635 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.542711 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:21.667241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.667178 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podStartSLOduration=2.667157324 podStartE2EDuration="2.667157324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:21.543225724 +0000 UTC m=+1789.154073540" watchObservedRunningTime="2026-11-03 21:45:21.667157324 +0000 UTC m=+1789.278005141" Nov 03 21:45:21.667784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.667761 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f"] Nov 03 21:45:21.674796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.674766 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-89ddad-29895705-6ps6f"] Nov 03 21:45:21.781700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.781649 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podStartSLOduration=2.781604334 podStartE2EDuration="2.781604334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:21.779517492 +0000 UTC m=+1789.390365308" watchObservedRunningTime="2026-11-03 21:45:21.781604334 +0000 UTC m=+1789.392452148" Nov 03 21:45:21.895036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.895001 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:21.905774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.905743 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:21.907259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.907234 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-bm7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:21.907382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.907287 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:45:21.918455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.918403 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:21.919572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.919550 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b89cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:21.919688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:21.919588 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:45:22.508294 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:22.508253 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" path="/var/lib/kubelet/pods/6cf8a456-7bf5-4ac9-90f7-3e5ed717c857/volumes" Nov 03 21:45:22.906118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:22.906079 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-bm7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:22.906300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:22.906130 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:45:22.919189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:22.919160 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b89cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:22.919325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:22.919217 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:45:23.906414 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:23.906377 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-bm7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:23.906834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:23.906427 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:45:23.918171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:23.918149 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b89cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:23.918266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:23.918195 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:45:24.906328 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:24.906292 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-bm7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:24.906502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:24.906341 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:45:24.918794 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:24.918677 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b89cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:24.918794 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:24.918732 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:45:25.906879 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:25.906840 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-bm7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:25.907383 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:25.906916 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:45:25.918204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:25.918161 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b89cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:25.918331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:25.918241 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:45:26.246965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:26.246878 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:26.906207 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:26.906168 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-bm7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:26.906394 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:26.906222 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:45:26.918406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:26.918373 5931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-b89cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:26.918883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:26.918442 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:45:27.105166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:27.105119 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:27.996908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:27.996839 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:28.262762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.262556 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:28.505133 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.505098 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:28.626809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.626343 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:28.626809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.626748 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:28.627646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.627596 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:45:28.688913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.688885 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:28.689192 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.689169 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:28.689932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:28.689914 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:45:29.381377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:29.381344 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:30.012221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:30.012176 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:30.343774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:30.343741 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:31.456721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:31.456669 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:33.067681 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:45:33.067650 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:45:34.906138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:34.906110 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:35.161118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:35.161053 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:35.519208 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:35.519137 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:45:35.527333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:35.527314 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:45:36.536457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:36.536425 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:36.831321 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:36.831240 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:39.770519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:39.770483 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:39.998636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:39.998588 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:40.087746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.087665 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9"] Nov 03 21:45:40.088344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088320 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088345 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088378 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088386 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088403 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088410 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088571 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088583 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.088747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.088709 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf8a456-7bf5-4ac9-90f7-3e5ed717c857" containerName="cleanup" Nov 03 21:45:40.093651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.093632 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.156252 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.156217 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9"] Nov 03 21:45:40.203683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.203639 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8677fbc4-52ea-4b94-ba97-571b0bbfc563-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.203894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.203759 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5ls\" (UniqueName: \"kubernetes.io/projected/8677fbc4-52ea-4b94-ba97-571b0bbfc563-kube-api-access-dp5ls\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.204645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.204518 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1c3bed\"" Nov 03 21:45:40.204645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.204576 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1c3bed-dockercfg-9m5jh\"" Nov 03 21:45:40.273845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.273805 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8"] Nov 03 21:45:40.278064 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.278043 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.304533 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.304508 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8677fbc4-52ea-4b94-ba97-571b0bbfc563-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.304691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.304671 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5ls\" (UniqueName: \"kubernetes.io/projected/8677fbc4-52ea-4b94-ba97-571b0bbfc563-kube-api-access-dp5ls\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.305344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.305319 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8677fbc4-52ea-4b94-ba97-571b0bbfc563-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.308729 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.308706 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8"] Nov 03 21:45:40.361258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.361231 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5ls\" (UniqueName: \"kubernetes.io/projected/8677fbc4-52ea-4b94-ba97-571b0bbfc563-kube-api-access-dp5ls\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.405406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.405381 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjz4n\" (UniqueName: \"kubernetes.io/projected/6f78311f-1b49-4e1a-b50d-1cf495bd7692-kube-api-access-bjz4n\") pod \"payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.405530 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.405496 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f78311f-1b49-4e1a-b50d-1cf495bd7692-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.410009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.409986 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:40.506639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.506580 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f78311f-1b49-4e1a-b50d-1cf495bd7692-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.509767 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.509249 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f78311f-1b49-4e1a-b50d-1cf495bd7692-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.510896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.510790 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjz4n\" (UniqueName: \"kubernetes.io/projected/6f78311f-1b49-4e1a-b50d-1cf495bd7692-kube-api-access-bjz4n\") pod \"payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.567312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.567281 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjz4n\" (UniqueName: \"kubernetes.io/projected/6f78311f-1b49-4e1a-b50d-1cf495bd7692-kube-api-access-bjz4n\") pod \"payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.594871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.594847 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:40.815404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:40.815381 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9"] Nov 03 21:45:40.817329 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:40.817301 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8677fbc4_52ea_4b94_ba97_571b0bbfc563.slice/crio-0da5638cf5f1ed183db1284c98be1b8f2f8008d672ad23941366f7f3a73f21bb WatchSource:0}: Error finding container 0da5638cf5f1ed183db1284c98be1b8f2f8008d672ad23941366f7f3a73f21bb: Status 404 returned error can't find the container with id 0da5638cf5f1ed183db1284c98be1b8f2f8008d672ad23941366f7f3a73f21bb Nov 03 21:45:41.008967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.008943 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8"] Nov 03 21:45:41.011153 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:45:41.011127 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f78311f_1b49_4e1a_b50d_1cf495bd7692.slice/crio-c8b66543fd3406bd752066813861f1a7c0ae7d1c56e22b9ccf62efc18ae21a85 WatchSource:0}: Error finding container c8b66543fd3406bd752066813861f1a7c0ae7d1c56e22b9ccf62efc18ae21a85: Status 404 returned error can't find the container with id c8b66543fd3406bd752066813861f1a7c0ae7d1c56e22b9ccf62efc18ae21a85 Nov 03 21:45:41.563691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.563652 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" event={"ID":"6f78311f-1b49-4e1a-b50d-1cf495bd7692","Type":"ContainerStarted","Data":"af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c"} Nov 03 21:45:41.563871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.563720 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:41.563871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.563744 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" event={"ID":"6f78311f-1b49-4e1a-b50d-1cf495bd7692","Type":"ContainerStarted","Data":"c8b66543fd3406bd752066813861f1a7c0ae7d1c56e22b9ccf62efc18ae21a85"} Nov 03 21:45:41.565314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.565287 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" event={"ID":"8677fbc4-52ea-4b94-ba97-571b0bbfc563","Type":"ContainerStarted","Data":"b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c"} Nov 03 21:45:41.565433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.565332 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" event={"ID":"8677fbc4-52ea-4b94-ba97-571b0bbfc563","Type":"ContainerStarted","Data":"0da5638cf5f1ed183db1284c98be1b8f2f8008d672ad23941366f7f3a73f21bb"} Nov 03 21:45:41.565433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.565353 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:41.679144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.679093 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" podStartSLOduration=1.679079905 podStartE2EDuration="1.679079905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:41.677443985 +0000 UTC m=+1809.288291811" watchObservedRunningTime="2026-11-03 21:45:41.679079905 +0000 UTC m=+1809.289927720" Nov 03 21:45:41.763583 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.763555 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:41.795467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:41.795373 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" podStartSLOduration=1.7953155509999998 podStartE2EDuration="1.795315551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:41.792419641 +0000 UTC m=+1809.403267454" watchObservedRunningTime="2026-11-03 21:45:41.795315551 +0000 UTC m=+1809.406163368" Nov 03 21:45:46.009335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:46.009294 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:46.850906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:46.849752 5931 patch_prober.go:29] interesting pod/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": read tcp 10.134.0.2:51558->10.134.0.104:15021: read: connection reset by peer" start-of-body= Nov 03 21:45:46.850906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:46.849805 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": read tcp 10.134.0.2:51558->10.134.0.104:15021: read: connection reset by peer" Nov 03 21:45:47.015509 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:47.015432 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:47.608076 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:47.608041 5931 generic.go:356] "Generic (PLEG): container finished" podID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerID="a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e" exitCode=0 Nov 03 21:45:47.608273 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:47.608115 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" event={"ID":"892329ca-8914-4602-ae6a-88a3d8666cdf","Type":"ContainerDied","Data":"a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e"} Nov 03 21:45:47.608537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:47.608520 5931 scope.go:122] "RemoveContainer" containerID="a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e" Nov 03 21:45:47.609965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:47.609943 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:47.610026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:47.610015 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:47.610062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:47.610048 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:45:48.616182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:48.616138 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" event={"ID":"892329ca-8914-4602-ae6a-88a3d8666cdf","Type":"ContainerStarted","Data":"8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739"} Nov 03 21:45:49.255065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:49.255032 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:45:49.255238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:49.255169 5931 patch_prober.go:29] interesting pod/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:49.255238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:49.255208 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 21:45:50.255295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:50.255264 5931 patch_prober.go:29] interesting pod/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:50.255705 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:50.255315 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 21:45:50.955455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:50.955420 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:51.254802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:51.254727 5931 patch_prober.go:29] interesting pod/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:51.254802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:51.254775 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 21:45:51.734027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:51.733970 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:51.946922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:51.946885 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:52.186845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.186812 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:52.258051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.258010 5931 patch_prober.go:29] interesting pod/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:52.258354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.258319 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 21:45:52.396366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.396338 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:52.575046 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.574952 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:45:52.575046 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.575029 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:45:52.613292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.613268 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:52.888341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:52.888305 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:53.115882 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:53.115845 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:53.255686 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:53.255571 5931 patch_prober.go:29] interesting pod/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:53.255686 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:53.255671 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 21:45:54.258628 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:54.258584 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:45:54.259019 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:54.259000 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:45:54.259788 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:54.259769 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:45:56.449623 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:56.449578 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:56.770093 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:56.770026 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:57.027779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:57.027699 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:57.474591 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:57.474561 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:57.696513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:57.696486 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:59.769122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:59.769089 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:59.996157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:45:59.996105 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:00.086022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.085947 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst"] Nov 03 21:46:00.090275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.090258 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.148402 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.148375 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst"] Nov 03 21:46:00.157344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.157238 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-kube-api-access-r6kzk\") pod \"payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.157519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.157489 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.193871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.193833 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3c6597\"" Nov 03 21:46:00.194089 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.193980 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3c6597-dockercfg-pvplp\"" Nov 03 21:46:00.252082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.252048 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf"] Nov 03 21:46:00.256336 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.256315 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.258386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.258364 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-kube-api-access-r6kzk\") pod \"payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.258479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.258409 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.259055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.259038 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.300224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.300199 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf"] Nov 03 21:46:00.314453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.314433 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-kube-api-access-r6kzk\") pod \"payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.359101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.359080 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8gv9\" (UniqueName: \"kubernetes.io/projected/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-kube-api-access-v8gv9\") pod \"payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.359226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.359120 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.404121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.404100 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.461265 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.460511 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8gv9\" (UniqueName: \"kubernetes.io/projected/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-kube-api-access-v8gv9\") pod \"payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.461265 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.460574 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.461488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.461441 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.517984 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.517953 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8gv9\" (UniqueName: \"kubernetes.io/projected/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-kube-api-access-v8gv9\") pod \"payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.569845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.569816 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:00.779603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.779576 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst"] Nov 03 21:46:00.780801 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:00.780769 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1ffa2a_19aa_4953_94f9_e9043cf69c28.slice/crio-07648d39e02c742c6f4f2e270f806b26e978dafe45bd2abf220ec21dfcce1a8f WatchSource:0}: Error finding container 07648d39e02c742c6f4f2e270f806b26e978dafe45bd2abf220ec21dfcce1a8f: Status 404 returned error can't find the container with id 07648d39e02c742c6f4f2e270f806b26e978dafe45bd2abf220ec21dfcce1a8f Nov 03 21:46:00.855032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.855006 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" event={"ID":"7a1ffa2a-19aa-4953-94f9-e9043cf69c28","Type":"ContainerStarted","Data":"ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1"} Nov 03 21:46:00.855127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.855058 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" event={"ID":"7a1ffa2a-19aa-4953-94f9-e9043cf69c28","Type":"ContainerStarted","Data":"07648d39e02c742c6f4f2e270f806b26e978dafe45bd2abf220ec21dfcce1a8f"} Nov 03 21:46:00.855215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.855196 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:00.943695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.943673 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf"] Nov 03 21:46:00.945558 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:00.945531 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09d3c35_08bf_4dfd_b115_757ec3e92bbc.slice/crio-968e4dbefc4eadc370192fd65ae4be836dd36d50f19e9d9785dcf92098fff63f WatchSource:0}: Error finding container 968e4dbefc4eadc370192fd65ae4be836dd36d50f19e9d9785dcf92098fff63f: Status 404 returned error can't find the container with id 968e4dbefc4eadc370192fd65ae4be836dd36d50f19e9d9785dcf92098fff63f Nov 03 21:46:00.969224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:00.969176 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" podStartSLOduration=0.969164653 podStartE2EDuration="969.164653ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:00.966705297 +0000 UTC m=+1828.577553111" watchObservedRunningTime="2026-11-03 21:46:00.969164653 +0000 UTC m=+1828.580012467" Nov 03 21:46:01.864667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:01.864585 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" event={"ID":"a09d3c35-08bf-4dfd-b115-757ec3e92bbc","Type":"ContainerStarted","Data":"9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9"} Nov 03 21:46:01.865152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:01.865130 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:01.865258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:01.865159 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" event={"ID":"a09d3c35-08bf-4dfd-b115-757ec3e92bbc","Type":"ContainerStarted","Data":"968e4dbefc4eadc370192fd65ae4be836dd36d50f19e9d9785dcf92098fff63f"} Nov 03 21:46:01.978952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:01.978892 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" podStartSLOduration=1.978872386 podStartE2EDuration="1.978872386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:01.976706117 +0000 UTC m=+1829.587553934" watchObservedRunningTime="2026-11-03 21:46:01.978872386 +0000 UTC m=+1829.589720200" Nov 03 21:46:02.104290 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:02.104127 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:03.073455 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:03.073419 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:46:04.780462 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:04.780254 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:05.030717 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.029385 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:05.070517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.070483 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788"] Nov 03 21:46:05.074793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.074774 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.152657 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.152000 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788"] Nov 03 21:46:05.185638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.183359 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c7c113-5044-47c8-a2d3-baaf8baf4086-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.185638 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.183431 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzrv\" (UniqueName: \"kubernetes.io/projected/49c7c113-5044-47c8-a2d3-baaf8baf4086-kube-api-access-hrzrv\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.207232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.200332 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3c0bdb\"" Nov 03 21:46:05.210537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.209036 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3c0bdb-dockercfg-vkfsg\"" Nov 03 21:46:05.262929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.262898 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb"] Nov 03 21:46:05.267819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.267800 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.285226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.285178 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c7c113-5044-47c8-a2d3-baaf8baf4086-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.285226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.285221 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzrv\" (UniqueName: \"kubernetes.io/projected/49c7c113-5044-47c8-a2d3-baaf8baf4086-kube-api-access-hrzrv\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.285859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.285841 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c7c113-5044-47c8-a2d3-baaf8baf4086-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.309526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.309506 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb"] Nov 03 21:46:05.350713 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.350656 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzrv\" (UniqueName: \"kubernetes.io/projected/49c7c113-5044-47c8-a2d3-baaf8baf4086-kube-api-access-hrzrv\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.386455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.386429 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ctf\" (UniqueName: \"kubernetes.io/projected/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-kube-api-access-d7ctf\") pod \"payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.386597 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.386585 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.389122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.389103 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.489431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.487587 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ctf\" (UniqueName: \"kubernetes.io/projected/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-kube-api-access-d7ctf\") pod \"payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.489431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.487852 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.489431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.488821 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.549651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.549565 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ctf\" (UniqueName: \"kubernetes.io/projected/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-kube-api-access-d7ctf\") pod \"payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.581281 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.581255 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:05.790095 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.790069 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788"] Nov 03 21:46:05.791969 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:05.791943 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c7c113_5044_47c8_a2d3_baaf8baf4086.slice/crio-50e0bbf4a3f3cfeffb4dfce61b372885b8db69d1311a86fff210a2d4cd4adef3 WatchSource:0}: Error finding container 50e0bbf4a3f3cfeffb4dfce61b372885b8db69d1311a86fff210a2d4cd4adef3: Status 404 returned error can't find the container with id 50e0bbf4a3f3cfeffb4dfce61b372885b8db69d1311a86fff210a2d4cd4adef3 Nov 03 21:46:05.893689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.893662 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" event={"ID":"49c7c113-5044-47c8-a2d3-baaf8baf4086","Type":"ContainerStarted","Data":"f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae"} Nov 03 21:46:05.893809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.893716 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" event={"ID":"49c7c113-5044-47c8-a2d3-baaf8baf4086","Type":"ContainerStarted","Data":"50e0bbf4a3f3cfeffb4dfce61b372885b8db69d1311a86fff210a2d4cd4adef3"} Nov 03 21:46:05.893809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.893742 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:05.982777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:05.982756 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb"] Nov 03 21:46:05.984459 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:05.984436 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80afdb19_0c01_4bb2_8d37_ec9e0bf20d7e.slice/crio-c7210d9d062579905e2b5c4042df00ebb595bfb315689d0d2d938bae819d7938 WatchSource:0}: Error finding container c7210d9d062579905e2b5c4042df00ebb595bfb315689d0d2d938bae819d7938: Status 404 returned error can't find the container with id c7210d9d062579905e2b5c4042df00ebb595bfb315689d0d2d938bae819d7938 Nov 03 21:46:06.008228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:06.008160 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" podStartSLOduration=1.008143459 podStartE2EDuration="1.008143459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:06.005747742 +0000 UTC m=+1833.616595557" watchObservedRunningTime="2026-11-03 21:46:06.008143459 +0000 UTC m=+1833.618991277" Nov 03 21:46:06.441663 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:06.441411 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:06.910132 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:06.910085 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" event={"ID":"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e","Type":"ContainerStarted","Data":"3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83"} Nov 03 21:46:06.910605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:06.910172 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" event={"ID":"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e","Type":"ContainerStarted","Data":"c7210d9d062579905e2b5c4042df00ebb595bfb315689d0d2d938bae819d7938"} Nov 03 21:46:06.910605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:06.910398 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:07.043671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:07.043594 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" podStartSLOduration=2.04357631 podStartE2EDuration="2.04357631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:07.042048963 +0000 UTC m=+1834.652896775" watchObservedRunningTime="2026-11-03 21:46:07.04357631 +0000 UTC m=+1834.654424126" Nov 03 21:46:07.947170 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:07.947137 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:08.176715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:08.176669 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:08.408233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:08.408202 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:08.598579 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:08.598549 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5"] Nov 03 21:46:08.598855 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:08.598827 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" containerID="cri-o://891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e" gracePeriod=30 Nov 03 21:46:08.838766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:08.838686 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:09.481153 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:09.481112 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr"] Nov 03 21:46:09.481682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:09.481370 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" containerID="cri-o://ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321" gracePeriod=30 Nov 03 21:46:09.588682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:09.588139 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:46:09.588682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:09.588212 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:46:09.764181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:09.764099 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:10.016926 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:10.016830 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:11.867673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:11.867648 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:46:11.945848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:11.945788 5931 generic.go:356] "Generic (PLEG): container finished" podID="972de09d-80a9-4a4c-b226-53933aae736f" containerID="0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b" exitCode=0 Nov 03 21:46:11.945971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:11.945854 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" event={"ID":"972de09d-80a9-4a4c-b226-53933aae736f","Type":"ContainerDied","Data":"0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b"} Nov 03 21:46:11.946174 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:11.946161 5931 scope.go:122] "RemoveContainer" containerID="0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b" Nov 03 21:46:11.947417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:11.947394 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:46:11.947471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:11.947456 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:46:11.947513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:11.947502 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:46:12.874496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:12.874468 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:46:12.956628 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:12.956574 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" event={"ID":"972de09d-80a9-4a4c-b226-53933aae736f","Type":"ContainerStarted","Data":"ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0"} Nov 03 21:46:13.098290 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.098251 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:46:13.149841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.149636 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:13.236902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.236830 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:46:13.241738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.241709 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:46:13.241958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.241934 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 21:46:13.611811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.611777 5931 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:46:13.611995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.611838 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:46:13.912221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.912200 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:46:13.964675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.964646 5931 generic.go:356] "Generic (PLEG): container finished" podID="572ab12d-96b0-40dd-91c3-67757727d099" containerID="891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e" exitCode=0 Nov 03 21:46:13.964819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.964688 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" Nov 03 21:46:13.964819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.964717 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" event={"ID":"572ab12d-96b0-40dd-91c3-67757727d099","Type":"ContainerDied","Data":"891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e"} Nov 03 21:46:13.964819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.964761 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5" event={"ID":"572ab12d-96b0-40dd-91c3-67757727d099","Type":"ContainerDied","Data":"91761efb63783a398040f0dc3d0c014be8e0eb4b3674da497261d9be12e4b03a"} Nov 03 21:46:13.964819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.964782 5931 scope.go:122] "RemoveContainer" containerID="891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e" Nov 03 21:46:13.974358 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.974341 5931 scope.go:122] "RemoveContainer" containerID="891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e" Nov 03 21:46:13.974639 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:13.974606 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e\": container with ID starting with 891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e not found: ID does not exist" containerID="891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e" Nov 03 21:46:13.974690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:13.974647 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e"} err="failed to get container status \"891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e\": rpc error: code = NotFound desc = could not find container \"891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e\": container with ID starting with 891af10bee37a040a864148c5f2657408e67cb79c6a12f6a134bacbe8f73d20e not found: ID does not exist" Nov 03 21:46:14.009750 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.009730 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-envoy\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.009841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.009757 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-data\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.009841 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.009819 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/572ab12d-96b0-40dd-91c3-67757727d099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572ab12d-96b0-40dd-91c3-67757727d099-istio-podinfo\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.009969 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.009843 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-certs\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.010025 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.009992 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-istio-token\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-istio-token\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.010098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010046 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-socket\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.010158 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010133 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-credential-socket\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.010247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010229 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-kube-api-access-2ld8c\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-kube-api-access-2ld8c\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.010312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010287 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-certs" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.010367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010347 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572ab12d-96b0-40dd-91c3-67757727d099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572ab12d-96b0-40dd-91c3-67757727d099-istiod-ca-cert\") pod \"572ab12d-96b0-40dd-91c3-67757727d099\" (UID: \"572ab12d-96b0-40dd-91c3-67757727d099\") " Nov 03 21:46:14.010459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010440 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-socket" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.010522 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010450 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-credential-socket" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.010781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010696 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-data" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.010951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.010895 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572ab12d-96b0-40dd-91c3-67757727d099-istiod-ca-cert" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:14.011421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.011130 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.011421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.011315 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.011421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.011336 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.011421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.011350 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.012123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.012102 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-envoy" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.012671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.012653 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-istio-token" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:14.012757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.012725 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-kube-api-access-2ld8c" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "kube-api-access-2ld8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:14.013010 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.012990 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/572ab12d-96b0-40dd-91c3-67757727d099-istio-podinfo" pod "572ab12d-96b0-40dd-91c3-67757727d099" (UID: "572ab12d-96b0-40dd-91c3-67757727d099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:46:14.112305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.112271 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572ab12d-96b0-40dd-91c3-67757727d099-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.112305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.112297 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572ab12d-96b0-40dd-91c3-67757727d099-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.112305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.112307 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572ab12d-96b0-40dd-91c3-67757727d099-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.112593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.112317 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.112593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.112325 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ld8c\" (UniqueName: \"kubernetes.io/projected/572ab12d-96b0-40dd-91c3-67757727d099-kube-api-access-2ld8c\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.243070 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.242980 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:46:14.243319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.243288 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 21:46:14.408056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.408025 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5"] Nov 03 21:46:14.413709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.413688 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-wsvj5"] Nov 03 21:46:14.506580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.506552 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572ab12d-96b0-40dd-91c3-67757727d099" path="/var/lib/kubelet/pods/572ab12d-96b0-40dd-91c3-67757727d099/volumes" Nov 03 21:46:14.803172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.803150 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:46:14.815605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.815579 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:14.921270 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921243 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-token\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-token\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921310 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istiod-ca-cert\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921472 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-data\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921576 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-kube-api-access-vqv6s\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-kube-api-access-vqv6s\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921652 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-socket\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921739 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-data" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.921949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921768 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-envoy\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921805 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-podinfo\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921846 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-credential-socket\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921878 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-certs\") pod \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\" (UID: \"eaec8e7a-caa1-4f31-953c-800b49b5f7c0\") " Nov 03 21:46:14.921949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.921937 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-socket" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.953878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.922416 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.953878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.922442 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:14.953878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.934648 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istiod-ca-cert" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:14.953878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.938689 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-certs" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.953878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.938711 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-credential-socket" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.953878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.953778 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-kube-api-access-vqv6s" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "kube-api-access-vqv6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:14.954431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.954106 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-podinfo" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:46:14.955314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.955290 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-envoy" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:14.955801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.955767 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-token" pod "eaec8e7a-caa1-4f31-953c-800b49b5f7c0" (UID: "eaec8e7a-caa1-4f31-953c-800b49b5f7c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:14.974668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.974644 5931 generic.go:356] "Generic (PLEG): container finished" podID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerID="ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321" exitCode=0 Nov 03 21:46:14.974772 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.974697 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" Nov 03 21:46:14.974772 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.974723 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" event={"ID":"eaec8e7a-caa1-4f31-953c-800b49b5f7c0","Type":"ContainerDied","Data":"ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321"} Nov 03 21:46:14.974857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.974774 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr" event={"ID":"eaec8e7a-caa1-4f31-953c-800b49b5f7c0","Type":"ContainerDied","Data":"0a6ddd458fd1bc28b849066fc5abfb7430392241a3dc9f4f574043a6e2d823b9"} Nov 03 21:46:14.974857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.974804 5931 scope.go:122] "RemoveContainer" containerID="ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321" Nov 03 21:46:14.996080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.991851 5931 scope.go:122] "RemoveContainer" containerID="ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321" Nov 03 21:46:14.998688 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:14.996360 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321\": container with ID starting with ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321 not found: ID does not exist" containerID="ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321" Nov 03 21:46:14.998688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:14.998644 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321"} err="failed to get container status \"ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321\": rpc error: code = NotFound desc = could not find container \"ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321\": container with ID starting with ade570757413af1db7864674285e8ccf0e71f1d1bdce2b9da50584bf00039321 not found: ID does not exist" Nov 03 21:46:15.047486 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.047418 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:15.047486 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.047444 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:15.047486 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.047459 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqv6s\" (UniqueName: \"kubernetes.io/projected/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-kube-api-access-vqv6s\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:15.047486 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.047473 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:15.047486 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.047487 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:15.047787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.047499 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:15.047787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.047512 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaec8e7a-caa1-4f31-953c-800b49b5f7c0-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:46:15.142646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.140107 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:15.159639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.159600 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr"] Nov 03 21:46:15.164985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.164961 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-p7zdr"] Nov 03 21:46:15.230953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.230925 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:46:15.231157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.231129 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 21:46:15.680637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:15.676370 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:16.236942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:16.236908 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:46:16.237492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:16.237463 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 21:46:16.506804 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:16.506735 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" path="/var/lib/kubelet/pods/eaec8e7a-caa1-4f31-953c-800b49b5f7c0/volumes" Nov 03 21:46:16.914102 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:16.914069 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:46:17.241685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.228063 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:46:17.241685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.228118 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 21:46:17.697684 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.697641 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p"] Nov 03 21:46:17.698224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.698206 5931 cpu_manager.go:414] "containerMap: removing container" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" Nov 03 21:46:17.698224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.698221 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" Nov 03 21:46:17.698393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.698239 5931 cpu_manager.go:414] "containerMap: removing container" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" Nov 03 21:46:17.698393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.698244 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" Nov 03 21:46:17.698393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.698334 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="572ab12d-96b0-40dd-91c3-67757727d099" containerName="istio-proxy" Nov 03 21:46:17.698393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.698345 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaec8e7a-caa1-4f31-953c-800b49b5f7c0" containerName="istio-proxy" Nov 03 21:46:17.703211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.703189 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.754673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.754645 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1c3bed-serving-cert\"" Nov 03 21:46:17.780880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.780851 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.781071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.780890 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da556241-a4a1-4b71-bac1-c3671f91a415-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.781071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.781036 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-metrics-certs\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.781071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.781062 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98xg\" (UniqueName: \"kubernetes.io/projected/da556241-a4a1-4b71-bac1-c3671f91a415-kube-api-access-n98xg\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.807064 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.807037 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p"] Nov 03 21:46:17.811037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.810986 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:17.882116 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.882086 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da556241-a4a1-4b71-bac1-c3671f91a415-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.882278 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.882242 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-metrics-certs\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.882317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.882280 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98xg\" (UniqueName: \"kubernetes.io/projected/da556241-a4a1-4b71-bac1-c3671f91a415-kube-api-access-n98xg\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.882426 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.882408 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.882582 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:17.882559 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-serving-cert: secret "maas-api-e2e-ipp-a-1c3bed-serving-cert" not found Nov 03 21:46:17.882663 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:17.882650 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls podName:da556241-a4a1-4b71-bac1-c3671f91a415 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:18.382630118 +0000 UTC m=+1845.993477911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls") pod "maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" (UID: "da556241-a4a1-4b71-bac1-c3671f91a415") : secret "maas-api-e2e-ipp-a-1c3bed-serving-cert" not found Nov 03 21:46:17.882820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.882801 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da556241-a4a1-4b71-bac1-c3671f91a415-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.884949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.884925 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-metrics-certs\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:17.920054 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.920036 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:46:17.939860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:17.939838 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98xg\" (UniqueName: \"kubernetes.io/projected/da556241-a4a1-4b71-bac1-c3671f91a415-kube-api-access-n98xg\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:18.059163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.059081 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:18.231812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.231782 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:46:18.232869 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.232851 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:46:18.282671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.282639 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv"] Nov 03 21:46:18.286842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.286821 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.294804 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.294703 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:18.346888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.346858 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv"] Nov 03 21:46:18.388831 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.388801 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-podinfo\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.388975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.388832 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-data\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.388975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.388863 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-envoy\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.389061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.389003 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-certs\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.389107 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.389074 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:18.389107 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.389102 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-socket\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.389182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.389143 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-token\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.389182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.389168 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7cvp\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-kube-api-access-m7cvp\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.389267 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:18.389193 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-serving-cert: secret "maas-api-e2e-ipp-a-1c3bed-serving-cert" not found Nov 03 21:46:18.389267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.389196 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-credential-socket\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.389267 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:18.389260 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls podName:da556241-a4a1-4b71-bac1-c3671f91a415 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:19.389241211 +0000 UTC m=+1847.000089018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls") pod "maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" (UID: "da556241-a4a1-4b71-bac1-c3671f91a415") : secret "maas-api-e2e-ipp-a-1c3bed-serving-cert" not found Nov 03 21:46:18.389380 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.389345 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9655b3a-b651-41f8-91e9-3ab609c61a79-istiod-ca-cert\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.391371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.391356 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3940f8-openshift-default-dockercfg-pjt5t\"" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491407 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-certs\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491537 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-socket\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491628 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-token\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491663 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7cvp\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-kube-api-access-m7cvp\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491703 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-credential-socket\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491755 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9655b3a-b651-41f8-91e9-3ab609c61a79-istiod-ca-cert\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491830 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-podinfo\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491857 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-data\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.491912 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-envoy\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.492996 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-certs\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.493229 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-socket\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.494065 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9655b3a-b651-41f8-91e9-3ab609c61a79-istiod-ca-cert\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.494290 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-credential-socket\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.497636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.494996 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-data\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.519196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.519165 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-envoy\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.522407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.522371 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-podinfo\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.534796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.534760 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:18.548639 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.548600 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-token\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.548725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.548660 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7cvp\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-kube-api-access-m7cvp\") pod \"e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.600867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.600802 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:18.782236 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.782203 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:18.998230 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:18.998153 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:19.039073 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.039043 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv"] Nov 03 21:46:19.041219 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:19.041192 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9655b3a_b651_41f8_91e9_3ab609c61a79.slice/crio-3cab56b5185dd484262160b20c0c0dabf50575ed04bf5dd443dffc033d0169f8 WatchSource:0}: Error finding container 3cab56b5185dd484262160b20c0c0dabf50575ed04bf5dd443dffc033d0169f8: Status 404 returned error can't find the container with id 3cab56b5185dd484262160b20c0c0dabf50575ed04bf5dd443dffc033d0169f8 Nov 03 21:46:19.043589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.043561 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:46:19.043689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.043663 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:46:19.043732 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.043697 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:46:19.404290 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.404258 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:19.407231 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.407200 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:19.517988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.517961 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:19.899471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:19.899444 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p"] Nov 03 21:46:19.902183 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:19.902156 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda556241_a4a1_4b71_bac1_c3671f91a415.slice/crio-95c7709acc1da44b7d8898795a2b354c907bb658ebc86378976cb64227293531 WatchSource:0}: Error finding container 95c7709acc1da44b7d8898795a2b354c907bb658ebc86378976cb64227293531: Status 404 returned error can't find the container with id 95c7709acc1da44b7d8898795a2b354c907bb658ebc86378976cb64227293531 Nov 03 21:46:20.012435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:20.012410 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" event={"ID":"a9655b3a-b651-41f8-91e9-3ab609c61a79","Type":"ContainerStarted","Data":"bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37"} Nov 03 21:46:20.012545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:20.012461 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" event={"ID":"a9655b3a-b651-41f8-91e9-3ab609c61a79","Type":"ContainerStarted","Data":"3cab56b5185dd484262160b20c0c0dabf50575ed04bf5dd443dffc033d0169f8"} Nov 03 21:46:20.013692 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:20.013670 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" event={"ID":"da556241-a4a1-4b71-bac1-c3671f91a415","Type":"ContainerStarted","Data":"95c7709acc1da44b7d8898795a2b354c907bb658ebc86378976cb64227293531"} Nov 03 21:46:20.144980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:20.144932 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podStartSLOduration=2.144918735 podStartE2EDuration="2.144918735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:20.142287614 +0000 UTC m=+1847.753135430" watchObservedRunningTime="2026-11-03 21:46:20.144918735 +0000 UTC m=+1847.755766550" Nov 03 21:46:20.603637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:20.602131 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:20.608745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:20.608715 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:21.021330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:21.021299 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" event={"ID":"da556241-a4a1-4b71-bac1-c3671f91a415","Type":"ContainerStarted","Data":"3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4"} Nov 03 21:46:21.021484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:21.021448 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:21.021484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:21.021477 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:21.022498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:21.022476 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:46:21.147950 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:21.147857 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" podStartSLOduration=3.263436574 podStartE2EDuration="4.147839937s" totalImagesPullingTime="884.403363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:21.144521447 +0000 UTC m=+1848.755369263" watchObservedRunningTime="2026-11-03 21:46:21.147839937 +0000 UTC m=+1848.758687759" Nov 03 21:46:23.144550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:23.144517 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:23.473082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:23.473016 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:23.712587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:23.712554 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:26.740861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:26.740829 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:26.982287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:26.982219 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:27.033822 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.033748 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:46:27.054830 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.054802 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l"] Nov 03 21:46:27.059193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.059172 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.111782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.111758 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3940f8\"" Nov 03 21:46:27.136266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.136241 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3940f8-dockercfg-c7qhf\"" Nov 03 21:46:27.140551 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.140525 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l"] Nov 03 21:46:27.217962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.217660 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7"] Nov 03 21:46:27.223276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.223241 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5g8d\" (UniqueName: \"kubernetes.io/projected/7a943f80-2717-4d0b-8484-cc52383809ba-kube-api-access-l5g8d\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.223445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.223420 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a943f80-2717-4d0b-8484-cc52383809ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.224427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.224386 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.274135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.274102 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7"] Nov 03 21:46:27.324218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.324136 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngrs\" (UniqueName: \"kubernetes.io/projected/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-kube-api-access-jngrs\") pod \"payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.324218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.324193 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.324420 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.324400 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a943f80-2717-4d0b-8484-cc52383809ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.324671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.324651 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5g8d\" (UniqueName: \"kubernetes.io/projected/7a943f80-2717-4d0b-8484-cc52383809ba-kube-api-access-l5g8d\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.325050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.325028 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a943f80-2717-4d0b-8484-cc52383809ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.382141 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.382119 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5g8d\" (UniqueName: \"kubernetes.io/projected/7a943f80-2717-4d0b-8484-cc52383809ba-kube-api-access-l5g8d\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.425851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.425827 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngrs\" (UniqueName: \"kubernetes.io/projected/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-kube-api-access-jngrs\") pod \"payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.425945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.425862 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.426427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.426410 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.496967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.496947 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngrs\" (UniqueName: \"kubernetes.io/projected/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-kube-api-access-jngrs\") pod \"payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.542812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.542790 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:27.701642 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.700854 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:27.934590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:27.934564 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7"] Nov 03 21:46:27.935810 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:27.935784 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6643bb7_9eaa_4ad7_8ee2_f582d7241752.slice/crio-07c63a704190e046c455ec4a8e2dc3762ddb14d1f7d3a2b0f0d046a0d6a39150 WatchSource:0}: Error finding container 07c63a704190e046c455ec4a8e2dc3762ddb14d1f7d3a2b0f0d046a0d6a39150: Status 404 returned error can't find the container with id 07c63a704190e046c455ec4a8e2dc3762ddb14d1f7d3a2b0f0d046a0d6a39150 Nov 03 21:46:28.068284 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:28.068253 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" event={"ID":"d6643bb7-9eaa-4ad7-8ee2-f582d7241752","Type":"ContainerStarted","Data":"9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479"} Nov 03 21:46:28.068411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:28.068306 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" event={"ID":"d6643bb7-9eaa-4ad7-8ee2-f582d7241752","Type":"ContainerStarted","Data":"07c63a704190e046c455ec4a8e2dc3762ddb14d1f7d3a2b0f0d046a0d6a39150"} Nov 03 21:46:28.068486 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:28.068417 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:28.116407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:28.116386 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l"] Nov 03 21:46:28.117887 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:28.117867 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a943f80_2717_4d0b_8484_cc52383809ba.slice/crio-558039b3ddcaf8f4069347c324ed876b9d321b9c8a946aa1b78fccf969e44bd4 WatchSource:0}: Error finding container 558039b3ddcaf8f4069347c324ed876b9d321b9c8a946aa1b78fccf969e44bd4: Status 404 returned error can't find the container with id 558039b3ddcaf8f4069347c324ed876b9d321b9c8a946aa1b78fccf969e44bd4 Nov 03 21:46:28.182510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:28.182438 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" podStartSLOduration=1.18242656 podStartE2EDuration="1.18242656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:28.181043694 +0000 UTC m=+1855.791891508" watchObservedRunningTime="2026-11-03 21:46:28.18242656 +0000 UTC m=+1855.793274413" Nov 03 21:46:28.573676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:28.573510 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:29.076125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:29.076094 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" event={"ID":"7a943f80-2717-4d0b-8484-cc52383809ba","Type":"ContainerStarted","Data":"ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4"} Nov 03 21:46:29.076529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:29.076166 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" event={"ID":"7a943f80-2717-4d0b-8484-cc52383809ba","Type":"ContainerStarted","Data":"558039b3ddcaf8f4069347c324ed876b9d321b9c8a946aa1b78fccf969e44bd4"} Nov 03 21:46:29.076529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:29.076188 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:29.191284 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:29.191241 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" podStartSLOduration=2.191228875 podStartE2EDuration="2.191228875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:29.187463742 +0000 UTC m=+1856.798311556" watchObservedRunningTime="2026-11-03 21:46:29.191228875 +0000 UTC m=+1856.802076690" Nov 03 21:46:33.068247 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:46:33.068215 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:46:33.767593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:33.767567 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:33.996883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:33.996851 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:39.079891 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:39.079865 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:46:39.618110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:39.618082 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:39.846073 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:39.846045 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:40.090859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:40.090839 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:46:40.095335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:40.095315 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:40.348504 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:40.348423 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:43.283886 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.283842 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:43.512834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.512781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:43.755786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.755739 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9"] Nov 03 21:46:43.763290 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.763263 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:43.788374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.788336 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc52c97f-0de4-4af8-acb3-7da8534e527c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:43.788625 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.788498 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgp8\" (UniqueName: \"kubernetes.io/projected/fc52c97f-0de4-4af8-acb3-7da8534e527c-kube-api-access-6hgp8\") pod \"payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:43.813264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.813235 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9"] Nov 03 21:46:43.858018 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.857982 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6ddbba-dockercfg-k9n5d\"" Nov 03 21:46:43.858018 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.857981 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6ddbba\"" Nov 03 21:46:43.890118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.890082 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgp8\" (UniqueName: \"kubernetes.io/projected/fc52c97f-0de4-4af8-acb3-7da8534e527c-kube-api-access-6hgp8\") pod \"payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:43.890259 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.890156 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc52c97f-0de4-4af8-acb3-7da8534e527c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:43.890851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.890833 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc52c97f-0de4-4af8-acb3-7da8534e527c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:43.920318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.920286 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr"] Nov 03 21:46:43.924537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.924522 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:43.946310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.946287 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgp8\" (UniqueName: \"kubernetes.io/projected/fc52c97f-0de4-4af8-acb3-7da8534e527c-kube-api-access-6hgp8\") pod \"payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:43.979626 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.979590 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr"] Nov 03 21:46:43.991460 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.991437 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55wp\" (UniqueName: \"kubernetes.io/projected/3addd93b-3e66-469f-bd37-267517de0347-kube-api-access-l55wp\") pod \"payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:43.991580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:43.991564 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3addd93b-3e66-469f-bd37-267517de0347-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:44.080860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.080803 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:44.092433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.092412 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55wp\" (UniqueName: \"kubernetes.io/projected/3addd93b-3e66-469f-bd37-267517de0347-kube-api-access-l55wp\") pod \"payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:44.092541 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.092523 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3addd93b-3e66-469f-bd37-267517de0347-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:44.093376 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.093349 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3addd93b-3e66-469f-bd37-267517de0347-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:44.157097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.157033 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55wp\" (UniqueName: \"kubernetes.io/projected/3addd93b-3e66-469f-bd37-267517de0347-kube-api-access-l55wp\") pod \"payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:44.238109 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.238074 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:44.445769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.445730 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:44.457573 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.457547 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9"] Nov 03 21:46:44.459461 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:44.459435 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc52c97f_0de4_4af8_acb3_7da8534e527c.slice/crio-492d324d762aa4b31fd4a3720ee1f9fb98242a7b9829b2af2ed79fa1985eff35 WatchSource:0}: Error finding container 492d324d762aa4b31fd4a3720ee1f9fb98242a7b9829b2af2ed79fa1985eff35: Status 404 returned error can't find the container with id 492d324d762aa4b31fd4a3720ee1f9fb98242a7b9829b2af2ed79fa1985eff35 Nov 03 21:46:44.618042 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.618022 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr"] Nov 03 21:46:44.619696 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:46:44.619675 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3addd93b_3e66_469f_bd37_267517de0347.slice/crio-c7ef8478920bd750057e149b2eb7f41a547dac4f779cfedc990e5098aa773b3c WatchSource:0}: Error finding container c7ef8478920bd750057e149b2eb7f41a547dac4f779cfedc990e5098aa773b3c: Status 404 returned error can't find the container with id c7ef8478920bd750057e149b2eb7f41a547dac4f779cfedc990e5098aa773b3c Nov 03 21:46:44.692836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.692802 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:44.922823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:44.922796 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:45.220811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.220735 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" event={"ID":"3addd93b-3e66-469f-bd37-267517de0347","Type":"ContainerStarted","Data":"e62164a1db8d4a6139c366daf6016ce5dd845547f4a3989140b4cd1328678c52"} Nov 03 21:46:45.220811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.220798 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" event={"ID":"3addd93b-3e66-469f-bd37-267517de0347","Type":"ContainerStarted","Data":"c7ef8478920bd750057e149b2eb7f41a547dac4f779cfedc990e5098aa773b3c"} Nov 03 21:46:45.221007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.220828 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:46:45.222427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.222407 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" event={"ID":"fc52c97f-0de4-4af8-acb3-7da8534e527c","Type":"ContainerStarted","Data":"ddde129ce44ad11bd1283ddc80c8d1c4896863d9fe66671f563e63873779e395"} Nov 03 21:46:45.222523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.222442 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" event={"ID":"fc52c97f-0de4-4af8-acb3-7da8534e527c","Type":"ContainerStarted","Data":"492d324d762aa4b31fd4a3720ee1f9fb98242a7b9829b2af2ed79fa1985eff35"} Nov 03 21:46:45.222523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.222468 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:45.336312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.336264 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" podStartSLOduration=2.336253813 podStartE2EDuration="2.336253813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:45.333438841 +0000 UTC m=+1872.944286656" watchObservedRunningTime="2026-11-03 21:46:45.336253813 +0000 UTC m=+1872.947101629" Nov 03 21:46:45.445021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:45.444912 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" podStartSLOduration=2.444896093 podStartE2EDuration="2.444896093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:45.444559419 +0000 UTC m=+1873.055407234" watchObservedRunningTime="2026-11-03 21:46:45.444896093 +0000 UTC m=+1873.055743910" Nov 03 21:46:46.345175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:46.345141 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:46.623505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:46.623431 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:49.867332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:49.867298 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:50.093135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:50.093105 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:50.315482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:50.315410 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:53.895400 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:53.895367 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:54.828115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:54.827686 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl"] Nov 03 21:46:54.828115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:54.827985 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" containerID="cri-o://c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618" gracePeriod=30 Nov 03 21:46:55.508499 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:55.508472 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:56.232429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:56.232403 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 21:46:56.232589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:46:56.232466 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 21:47:00.095640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.095602 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:47:00.164531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.164479 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:00.172370 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172351 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-certs\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172390 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-kube-api-access-664hj\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-kube-api-access-664hj\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172438 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-token\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-token\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172456 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istiod-ca-cert\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172521 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-envoy\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172550 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-data\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172660 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172601 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-credential-socket\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172660 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172637 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-certs" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:00.172764 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172732 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-podinfo\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172812 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-socket\") pod \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\" (UID: \"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f\") " Nov 03 21:47:00.172898 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172878 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-credential-socket" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:00.172994 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.172962 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istiod-ca-cert" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:00.173214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.173132 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-socket" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:00.173363 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.173240 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-data" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:00.173587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.173567 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.173656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.173596 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.173656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.173630 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.173656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.173645 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.173800 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.173659 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.174762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.174739 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-token" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:00.174864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.174790 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-kube-api-access-664hj" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "kube-api-access-664hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:00.174864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.174856 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-envoy" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:00.175344 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.175328 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-podinfo" pod "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" (UID: "8aaec1e3-3fc9-49e1-a1ec-1622ef13335f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:47:00.274773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.274746 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.274773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.274770 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.274934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.274781 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.274934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.274790 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664hj\" (UniqueName: \"kubernetes.io/projected/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f-kube-api-access-664hj\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:00.330422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.329407 5931 generic.go:356] "Generic (PLEG): container finished" podID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerID="c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618" exitCode=0 Nov 03 21:47:00.330422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.329671 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" event={"ID":"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f","Type":"ContainerDied","Data":"c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618"} Nov 03 21:47:00.330422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.329751 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" event={"ID":"8aaec1e3-3fc9-49e1-a1ec-1622ef13335f","Type":"ContainerDied","Data":"b1556082b4c1da8861ef98441511a197693a6e43fd5919f70a56be2fc7e2850b"} Nov 03 21:47:00.330422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.329791 5931 scope.go:122] "RemoveContainer" containerID="c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618" Nov 03 21:47:00.330422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.329991 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl" Nov 03 21:47:00.344224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.344205 5931 scope.go:122] "RemoveContainer" containerID="c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618" Nov 03 21:47:00.344491 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:47:00.344473 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618\": container with ID starting with c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618 not found: ID does not exist" containerID="c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618" Nov 03 21:47:00.344543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.344499 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618"} err="failed to get container status \"c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618\": rpc error: code = NotFound desc = could not find container \"c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618\": container with ID starting with c7c3e9252b8cc888246493015f92478ea3a2fe6bc213fb914301bcb4a615e618 not found: ID does not exist" Nov 03 21:47:00.409165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.409141 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:00.506789 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.506732 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl"] Nov 03 21:47:00.506789 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.506767 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-hrcgl"] Nov 03 21:47:00.830479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:00.830409 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:02.506900 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:02.506870 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" path="/var/lib/kubelet/pods/8aaec1e3-3fc9-49e1-a1ec-1622ef13335f/volumes" Nov 03 21:47:03.068970 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:47:03.068939 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:47:06.087024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:06.086986 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:10.874712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:10.874598 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:15.543867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:15.543834 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:19.887200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.887168 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf"] Nov 03 21:47:19.887807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.887793 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" Nov 03 21:47:19.887854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.887809 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" Nov 03 21:47:19.887933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.887924 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaec1e3-3fc9-49e1-a1ec-1622ef13335f" containerName="istio-proxy" Nov 03 21:47:19.892817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.892802 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:19.972353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.972320 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf5107-7bdd-474c-8717-6cbf5dffc680-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:19.972498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.972438 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtmm\" (UniqueName: \"kubernetes.io/projected/56cf5107-7bdd-474c-8717-6cbf5dffc680-kube-api-access-5qtmm\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:19.972576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.972516 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:19.972576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.972561 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-metrics-certs\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:19.994022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.993998 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3940f8-serving-cert\"" Nov 03 21:47:19.997096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:19.997073 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf"] Nov 03 21:47:20.073780 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.073744 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtmm\" (UniqueName: \"kubernetes.io/projected/56cf5107-7bdd-474c-8717-6cbf5dffc680-kube-api-access-5qtmm\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.073921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.073870 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.073968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.073923 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-metrics-certs\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.074020 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:47:20.074000 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-serving-cert: secret "maas-api-e2e-ipp-a-3940f8-serving-cert" not found Nov 03 21:47:20.074062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.074036 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf5107-7bdd-474c-8717-6cbf5dffc680-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.074107 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:47:20.074080 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls podName:56cf5107-7bdd-474c-8717-6cbf5dffc680 nodeName:}" failed. No retries permitted until 2026-11-03 21:47:20.574060966 +0000 UTC m=+1908.184908778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls") pod "maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" (UID: "56cf5107-7bdd-474c-8717-6cbf5dffc680") : secret "maas-api-e2e-ipp-a-3940f8-serving-cert" not found Nov 03 21:47:20.074685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.074663 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf5107-7bdd-474c-8717-6cbf5dffc680-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.076256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.076233 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-metrics-certs\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.138871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.138798 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtmm\" (UniqueName: \"kubernetes.io/projected/56cf5107-7bdd-474c-8717-6cbf5dffc680-kube-api-access-5qtmm\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.184794 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.184764 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:20.400427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.400354 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:20.580695 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.580664 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.590364 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.584368 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:20.612506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.612483 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:20.815538 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:20.815480 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:21.200667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:21.200642 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf"] Nov 03 21:47:21.202345 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:47:21.202318 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cf5107_7bdd_474c_8717_6cbf5dffc680.slice/crio-c281d9e9189edc1f3dbabedcbe298f0388719b4c8d57d1b5a229bba1ea4da5ba WatchSource:0}: Error finding container c281d9e9189edc1f3dbabedcbe298f0388719b4c8d57d1b5a229bba1ea4da5ba: Status 404 returned error can't find the container with id c281d9e9189edc1f3dbabedcbe298f0388719b4c8d57d1b5a229bba1ea4da5ba Nov 03 21:47:21.203629 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:21.203595 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:47:21.490216 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:21.490150 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" event={"ID":"56cf5107-7bdd-474c-8717-6cbf5dffc680","Type":"ContainerStarted","Data":"c281d9e9189edc1f3dbabedcbe298f0388719b4c8d57d1b5a229bba1ea4da5ba"} Nov 03 21:47:22.240945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:22.240908 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:22.506023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:22.505995 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" event={"ID":"56cf5107-7bdd-474c-8717-6cbf5dffc680","Type":"ContainerStarted","Data":"8dcb0b80dffeef8613dd4877717bf0fa109c4394c0ca2559cd549ba30df268fc"} Nov 03 21:47:22.506187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:22.506047 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:22.618289 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:22.618242 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" podStartSLOduration=2.771169647 podStartE2EDuration="3.618229684s" totalImagesPullingTime="847.060037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:22.616065991 +0000 UTC m=+1910.226913802" watchObservedRunningTime="2026-11-03 21:47:22.618229684 +0000 UTC m=+1910.229077498" Nov 03 21:47:23.193965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:23.193931 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:24.094589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:24.093722 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:24.556087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:24.555987 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:25.520727 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:25.520694 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:26.444139 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:26.444108 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8"] Nov 03 21:47:26.555306 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:26.555278 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9"] Nov 03 21:47:26.933676 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:26.933640 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:27.151735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:27.151698 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:28.140636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:28.140574 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:28.363605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:28.363572 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:28.510935 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:28.510851 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:47:30.593374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:30.593330 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:31.444565 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.444523 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" podUID="6f78311f-1b49-4e1a-b50d-1cf495bd7692" containerName="payload-processing" containerID="cri-o://af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c" gracePeriod=25 Nov 03 21:47:31.555817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.555778 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" podUID="8677fbc4-52ea-4b94-ba97-571b0bbfc563" containerName="payload-pre-processing" containerID="cri-o://b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c" gracePeriod=25 Nov 03 21:47:31.722855 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.722815 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:47:31.804378 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.804348 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f78311f-1b49-4e1a-b50d-1cf495bd7692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f78311f-1b49-4e1a-b50d-1cf495bd7692-plugins-config-volume\") pod \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " Nov 03 21:47:31.804559 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.804545 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f78311f-1b49-4e1a-b50d-1cf495bd7692-kube-api-access-bjz4n\" (UniqueName: \"kubernetes.io/projected/6f78311f-1b49-4e1a-b50d-1cf495bd7692-kube-api-access-bjz4n\") pod \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\" (UID: \"6f78311f-1b49-4e1a-b50d-1cf495bd7692\") " Nov 03 21:47:31.804715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.804695 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f78311f-1b49-4e1a-b50d-1cf495bd7692-plugins-config-volume" pod "6f78311f-1b49-4e1a-b50d-1cf495bd7692" (UID: "6f78311f-1b49-4e1a-b50d-1cf495bd7692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:31.805058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.805031 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f78311f-1b49-4e1a-b50d-1cf495bd7692-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:31.806652 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.806596 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f78311f-1b49-4e1a-b50d-1cf495bd7692-kube-api-access-bjz4n" pod "6f78311f-1b49-4e1a-b50d-1cf495bd7692" (UID: "6f78311f-1b49-4e1a-b50d-1cf495bd7692"). InnerVolumeSpecName "kube-api-access-bjz4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:31.824583 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.824563 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:47:31.906128 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.906105 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8677fbc4-52ea-4b94-ba97-571b0bbfc563-kube-api-access-dp5ls\" (UniqueName: \"kubernetes.io/projected/8677fbc4-52ea-4b94-ba97-571b0bbfc563-kube-api-access-dp5ls\") pod \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " Nov 03 21:47:31.906243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.906231 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8677fbc4-52ea-4b94-ba97-571b0bbfc563-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8677fbc4-52ea-4b94-ba97-571b0bbfc563-plugins-config-volume\") pod \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\" (UID: \"8677fbc4-52ea-4b94-ba97-571b0bbfc563\") " Nov 03 21:47:31.906581 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.906558 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8677fbc4-52ea-4b94-ba97-571b0bbfc563-plugins-config-volume" pod "8677fbc4-52ea-4b94-ba97-571b0bbfc563" (UID: "8677fbc4-52ea-4b94-ba97-571b0bbfc563"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:31.906875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.906717 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjz4n\" (UniqueName: \"kubernetes.io/projected/6f78311f-1b49-4e1a-b50d-1cf495bd7692-kube-api-access-bjz4n\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:31.906875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.906738 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8677fbc4-52ea-4b94-ba97-571b0bbfc563-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:31.908234 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:31.908214 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8677fbc4-52ea-4b94-ba97-571b0bbfc563-kube-api-access-dp5ls" pod "8677fbc4-52ea-4b94-ba97-571b0bbfc563" (UID: "8677fbc4-52ea-4b94-ba97-571b0bbfc563"). InnerVolumeSpecName "kube-api-access-dp5ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:32.008294 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.008236 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5ls\" (UniqueName: \"kubernetes.io/projected/8677fbc4-52ea-4b94-ba97-571b0bbfc563-kube-api-access-dp5ls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:47:32.563863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.563833 5931 generic.go:356] "Generic (PLEG): container finished" podID="6f78311f-1b49-4e1a-b50d-1cf495bd7692" containerID="af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c" exitCode=0 Nov 03 21:47:32.563981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.563901 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" Nov 03 21:47:32.564051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.563906 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" event={"ID":"6f78311f-1b49-4e1a-b50d-1cf495bd7692","Type":"ContainerDied","Data":"af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c"} Nov 03 21:47:32.564051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.564038 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8" event={"ID":"6f78311f-1b49-4e1a-b50d-1cf495bd7692","Type":"ContainerDied","Data":"c8b66543fd3406bd752066813861f1a7c0ae7d1c56e22b9ccf62efc18ae21a85"} Nov 03 21:47:32.564161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.564071 5931 scope.go:122] "RemoveContainer" containerID="af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c" Nov 03 21:47:32.565911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.565887 5931 generic.go:356] "Generic (PLEG): container finished" podID="8677fbc4-52ea-4b94-ba97-571b0bbfc563" containerID="b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c" exitCode=0 Nov 03 21:47:32.566010 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.565915 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" event={"ID":"8677fbc4-52ea-4b94-ba97-571b0bbfc563","Type":"ContainerDied","Data":"b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c"} Nov 03 21:47:32.566010 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.565978 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" event={"ID":"8677fbc4-52ea-4b94-ba97-571b0bbfc563","Type":"ContainerDied","Data":"0da5638cf5f1ed183db1284c98be1b8f2f8008d672ad23941366f7f3a73f21bb"} Nov 03 21:47:32.566182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.566157 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9" Nov 03 21:47:32.574247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.574231 5931 scope.go:122] "RemoveContainer" containerID="af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c" Nov 03 21:47:32.574493 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:47:32.574475 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c\": container with ID starting with af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c not found: ID does not exist" containerID="af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c" Nov 03 21:47:32.574567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.574502 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c"} err="failed to get container status \"af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c\": rpc error: code = NotFound desc = could not find container \"af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c\": container with ID starting with af5d95925381c1cb6fb8ae6e83fa7483b07ec5961e627ffb6544e38bcf4cab9c not found: ID does not exist" Nov 03 21:47:32.574567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.574524 5931 scope.go:122] "RemoveContainer" containerID="b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c" Nov 03 21:47:32.583361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.583335 5931 scope.go:122] "RemoveContainer" containerID="b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c" Nov 03 21:47:32.583606 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:47:32.583586 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c\": container with ID starting with b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c not found: ID does not exist" containerID="b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c" Nov 03 21:47:32.583700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.583638 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c"} err="failed to get container status \"b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c\": rpc error: code = NotFound desc = could not find container \"b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c\": container with ID starting with b8cad072f72a3ac65a9a7436572a2dcade9885211fb1909a9d9c8372bd65500c not found: ID does not exist" Nov 03 21:47:32.726205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.726157 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9"] Nov 03 21:47:32.775470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.775448 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-576c69999-fpbb9"] Nov 03 21:47:32.834661 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.834641 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8"] Nov 03 21:47:32.838488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:32.838465 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-68b64d84dd-vc8x8"] Nov 03 21:47:33.068403 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:47:33.068330 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:47:34.507070 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:34.507036 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f78311f-1b49-4e1a-b50d-1cf495bd7692" path="/var/lib/kubelet/pods/6f78311f-1b49-4e1a-b50d-1cf495bd7692/volumes" Nov 03 21:47:34.507428 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:34.507399 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8677fbc4-52ea-4b94-ba97-571b0bbfc563" path="/var/lib/kubelet/pods/8677fbc4-52ea-4b94-ba97-571b0bbfc563/volumes" Nov 03 21:47:35.368297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.368260 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:35.587103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.587071 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:35.868026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.867980 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh"] Nov 03 21:47:35.868954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.868920 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6f78311f-1b49-4e1a-b50d-1cf495bd7692" containerName="payload-processing" Nov 03 21:47:35.868954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.868946 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f78311f-1b49-4e1a-b50d-1cf495bd7692" containerName="payload-processing" Nov 03 21:47:35.869122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.869027 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8677fbc4-52ea-4b94-ba97-571b0bbfc563" containerName="payload-pre-processing" Nov 03 21:47:35.869122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.869035 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8677fbc4-52ea-4b94-ba97-571b0bbfc563" containerName="payload-pre-processing" Nov 03 21:47:35.869240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.869182 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8677fbc4-52ea-4b94-ba97-571b0bbfc563" containerName="payload-pre-processing" Nov 03 21:47:35.869240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.869198 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f78311f-1b49-4e1a-b50d-1cf495bd7692" containerName="payload-processing" Nov 03 21:47:35.881200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.881167 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:35.897925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.897893 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:35.916885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.916857 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh"] Nov 03 21:47:35.962371 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.962349 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-204763\"" Nov 03 21:47:35.962690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:35.962673 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-204763-dockercfg-pdrmz\"" Nov 03 21:47:36.017806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.017783 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt"] Nov 03 21:47:36.021828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.021806 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.076985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.076966 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt"] Nov 03 21:47:36.079429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.079403 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhgx\" (UniqueName: \"kubernetes.io/projected/cce2c9e8-6f22-4bbd-a08b-652366f03835-kube-api-access-kdhgx\") pod \"payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:36.079587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.079567 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce2c9e8-6f22-4bbd-a08b-652366f03835-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:36.180988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.180925 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce2c9e8-6f22-4bbd-a08b-652366f03835-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:36.181093 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.181001 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vwv\" (UniqueName: \"kubernetes.io/projected/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-kube-api-access-d6vwv\") pod \"payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.181093 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.181033 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.181186 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.181130 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhgx\" (UniqueName: \"kubernetes.io/projected/cce2c9e8-6f22-4bbd-a08b-652366f03835-kube-api-access-kdhgx\") pod \"payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:36.181554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.181534 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce2c9e8-6f22-4bbd-a08b-652366f03835-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:36.238006 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.237982 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhgx\" (UniqueName: \"kubernetes.io/projected/cce2c9e8-6f22-4bbd-a08b-652366f03835-kube-api-access-kdhgx\") pod \"payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:36.281905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.281881 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vwv\" (UniqueName: \"kubernetes.io/projected/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-kube-api-access-d6vwv\") pod \"payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.282000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.281920 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.282531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.282512 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.337920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.337899 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vwv\" (UniqueName: \"kubernetes.io/projected/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-kube-api-access-d6vwv\") pod \"payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.339248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.339235 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:36.511828 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.511759 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:36.714332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.714301 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt"] Nov 03 21:47:36.716392 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:47:36.716363 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc738b8a9_c20a_465a_b5bd_09bcf1d647fd.slice/crio-a1e918170703d35870794848032f4b4eb3d3a8d574bb89d30743817591c97449 WatchSource:0}: Error finding container a1e918170703d35870794848032f4b4eb3d3a8d574bb89d30743817591c97449: Status 404 returned error can't find the container with id a1e918170703d35870794848032f4b4eb3d3a8d574bb89d30743817591c97449 Nov 03 21:47:36.884775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:36.884752 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh"] Nov 03 21:47:36.885594 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:47:36.885570 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce2c9e8_6f22_4bbd_a08b_652366f03835.slice/crio-c6f6a623524ef40578b4717152ec9d478a85202ff098fbe65ec69a45b06d3f32 WatchSource:0}: Error finding container c6f6a623524ef40578b4717152ec9d478a85202ff098fbe65ec69a45b06d3f32: Status 404 returned error can't find the container with id c6f6a623524ef40578b4717152ec9d478a85202ff098fbe65ec69a45b06d3f32 Nov 03 21:47:37.614044 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.614011 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" event={"ID":"c738b8a9-c20a-465a-b5bd-09bcf1d647fd","Type":"ContainerStarted","Data":"e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de"} Nov 03 21:47:37.614240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.614082 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" event={"ID":"c738b8a9-c20a-465a-b5bd-09bcf1d647fd","Type":"ContainerStarted","Data":"a1e918170703d35870794848032f4b4eb3d3a8d574bb89d30743817591c97449"} Nov 03 21:47:37.614240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.614127 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:37.615749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.615725 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" event={"ID":"cce2c9e8-6f22-4bbd-a08b-652366f03835","Type":"ContainerStarted","Data":"747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954"} Nov 03 21:47:37.615900 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.615761 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" event={"ID":"cce2c9e8-6f22-4bbd-a08b-652366f03835","Type":"ContainerStarted","Data":"c6f6a623524ef40578b4717152ec9d478a85202ff098fbe65ec69a45b06d3f32"} Nov 03 21:47:37.615900 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.615825 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:37.727231 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.727189 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" podStartSLOduration=2.727177687 podStartE2EDuration="2.727177687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:37.725418792 +0000 UTC m=+1925.336266608" watchObservedRunningTime="2026-11-03 21:47:37.727177687 +0000 UTC m=+1925.338025501" Nov 03 21:47:37.843693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:37.843569 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" podStartSLOduration=2.843552124 podStartE2EDuration="2.843552124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:37.838019936 +0000 UTC m=+1925.448867753" watchObservedRunningTime="2026-11-03 21:47:37.843552124 +0000 UTC m=+1925.454399940" Nov 03 21:47:39.743925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:39.743893 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:39.953567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:39.953537 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:40.570819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:40.570788 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:40.788817 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:40.788783 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:40.996934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:40.996904 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:45.521594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:45.521541 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:46.636249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:46.636216 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:47.701829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:47.701789 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:48.004032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:48.003956 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:48.648322 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:48.648293 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:47:48.651749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:48.651710 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:47:50.328834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:50.328800 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:55.113957 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:55.113921 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:55.769942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:55.769873 5931 generic.go:356] "Generic (PLEG): container finished" podID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerID="bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37" exitCode=0 Nov 03 21:47:55.770125 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:55.769942 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" event={"ID":"a9655b3a-b651-41f8-91e9-3ab609c61a79","Type":"ContainerDied","Data":"bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37"} Nov 03 21:47:55.770292 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:55.770278 5931 scope.go:122] "RemoveContainer" containerID="bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37" Nov 03 21:47:55.771668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:55.771641 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:47:55.771774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:55.771722 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:47:55.771837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:55.771772 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:47:56.778553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:56.778523 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" event={"ID":"a9655b3a-b651-41f8-91e9-3ab609c61a79","Type":"ContainerStarted","Data":"1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa"} Nov 03 21:47:57.601124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:57.601090 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:47:57.601325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:57.601252 5931 patch_prober.go:29] interesting pod/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:47:57.601325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:57.601295 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:47:58.602130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:58.602104 5931 patch_prober.go:29] interesting pod/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:47:58.602501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:58.602148 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:47:59.602047 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:59.602016 5931 patch_prober.go:29] interesting pod/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:47:59.602243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:59.602070 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:47:59.867050 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:47:59.866968 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:00.079310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:00.079265 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:00.329085 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:00.329000 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:00.601411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:00.601376 5931 patch_prober.go:29] interesting pod/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:48:00.601574 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:00.601440 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:48:00.700766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:00.700723 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:01.391054 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:01.391018 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:01.601825 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:01.601795 5931 patch_prober.go:29] interesting pod/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:48:01.601987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:01.601849 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:48:01.879279 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:01.879229 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb"] Nov 03 21:48:01.985346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:01.985313 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788"] Nov 03 21:48:02.310880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.310747 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:02.526038 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.525973 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:02.606668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.606630 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:48:02.606919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.606899 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:48:02.607654 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.607603 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:48:02.742983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.742952 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt"] Nov 03 21:48:02.746011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.745990 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.797935 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.797908 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-62cf5c-openshift-default-dockercfg-rj4d9\"" Nov 03 21:48:02.806481 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.806457 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt"] Nov 03 21:48:02.823268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823234 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbczv\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-kube-api-access-qbczv\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823319 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-socket\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823479 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823463 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b67643-5e40-46ea-a541-8377a72de3a0-istio-podinfo\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823494 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b67643-5e40-46ea-a541-8377a72de3a0-istiod-ca-cert\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823593 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-data\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823717 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823649 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-certs\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823731 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-credential-socket\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823887 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-envoy\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.823969 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.823920 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-istio-token\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.924126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924048 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:02.924648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924628 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbczv\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-kube-api-access-qbczv\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.924741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924672 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-socket\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.924799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924775 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b67643-5e40-46ea-a541-8377a72de3a0-istio-podinfo\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.924852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924809 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b67643-5e40-46ea-a541-8377a72de3a0-istiod-ca-cert\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.924905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924860 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-data\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.924905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924896 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-certs\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.924930 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-credential-socket\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925089 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.925071 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-envoy\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.925100 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-istio-token\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.925137 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-socket\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925282 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.925263 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-data\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.925754 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-certs\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.925765 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-credential-socket\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.925963 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.925944 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b67643-5e40-46ea-a541-8377a72de3a0-istiod-ca-cert\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.927415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.927391 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b67643-5e40-46ea-a541-8377a72de3a0-istio-podinfo\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:02.927814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:02.927798 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-envoy\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:03.017661 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.013526 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-istio-token\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:03.022515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.022483 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbczv\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-kube-api-access-qbczv\") pod \"e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:03.061658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.058687 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:03.092135 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:03.092099 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:48:03.211866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.211794 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:03.497986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.497937 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt"] Nov 03 21:48:03.500737 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:48:03.500708 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b67643_5e40_46ea_a541_8377a72de3a0.slice/crio-eea9ace50e8324f05bd40b07a00fd0ce4441cb5caa82504e9894b9408cdda3cd WatchSource:0}: Error finding container eea9ace50e8324f05bd40b07a00fd0ce4441cb5caa82504e9894b9408cdda3cd: Status 404 returned error can't find the container with id eea9ace50e8324f05bd40b07a00fd0ce4441cb5caa82504e9894b9408cdda3cd Nov 03 21:48:03.503219 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.503186 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:48:03.503317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.503262 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:48:03.503317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.503305 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:48:03.833978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.833886 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" event={"ID":"11b67643-5e40-46ea-a541-8377a72de3a0","Type":"ContainerStarted","Data":"f747a8e5d524d436d6aaa126eaccc9ee2d488b3195135660bcff73e4e538289f"} Nov 03 21:48:03.833978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.833961 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" event={"ID":"11b67643-5e40-46ea-a541-8377a72de3a0","Type":"ContainerStarted","Data":"eea9ace50e8324f05bd40b07a00fd0ce4441cb5caa82504e9894b9408cdda3cd"} Nov 03 21:48:03.950706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:03.950652 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podStartSLOduration=1.95060744 podStartE2EDuration="1.95060744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:03.948879529 +0000 UTC m=+1951.559727350" watchObservedRunningTime="2026-11-03 21:48:03.95060744 +0000 UTC m=+1951.561455254" Nov 03 21:48:04.059852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:04.059821 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:04.074127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:04.074099 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:04.841382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:04.841354 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:04.842256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:04.842238 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:48:05.860892 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:05.860860 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:06.881055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:06.881022 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" podUID="80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" containerName="payload-processing" containerID="cri-o://3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83" gracePeriod=25 Nov 03 21:48:06.988105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:06.986465 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" podUID="49c7c113-5044-47c8-a2d3-baaf8baf4086" containerName="payload-pre-processing" containerID="cri-o://f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae" gracePeriod=25 Nov 03 21:48:07.147539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.147246 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:48:07.274975 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.274948 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:48:07.279006 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.278982 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-plugins-config-volume\") pod \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " Nov 03 21:48:07.279126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.279042 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-kube-api-access-d7ctf\" (UniqueName: \"kubernetes.io/projected/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-kube-api-access-d7ctf\") pod \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\" (UID: \"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e\") " Nov 03 21:48:07.279321 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.279299 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-plugins-config-volume" pod "80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" (UID: "80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:07.279512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.279486 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:07.281311 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.281287 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-kube-api-access-d7ctf" pod "80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" (UID: "80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e"). InnerVolumeSpecName "kube-api-access-d7ctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:07.380561 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.380536 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c7c113-5044-47c8-a2d3-baaf8baf4086-kube-api-access-hrzrv\" (UniqueName: \"kubernetes.io/projected/49c7c113-5044-47c8-a2d3-baaf8baf4086-kube-api-access-hrzrv\") pod \"49c7c113-5044-47c8-a2d3-baaf8baf4086\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " Nov 03 21:48:07.380728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.380606 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c7c113-5044-47c8-a2d3-baaf8baf4086-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c7c113-5044-47c8-a2d3-baaf8baf4086-plugins-config-volume\") pod \"49c7c113-5044-47c8-a2d3-baaf8baf4086\" (UID: \"49c7c113-5044-47c8-a2d3-baaf8baf4086\") " Nov 03 21:48:07.380898 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.380877 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c7c113-5044-47c8-a2d3-baaf8baf4086-plugins-config-volume" pod "49c7c113-5044-47c8-a2d3-baaf8baf4086" (UID: "49c7c113-5044-47c8-a2d3-baaf8baf4086"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:07.381002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.380990 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7ctf\" (UniqueName: \"kubernetes.io/projected/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e-kube-api-access-d7ctf\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:07.381045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.381004 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c7c113-5044-47c8-a2d3-baaf8baf4086-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:07.382594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.382574 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c7c113-5044-47c8-a2d3-baaf8baf4086-kube-api-access-hrzrv" pod "49c7c113-5044-47c8-a2d3-baaf8baf4086" (UID: "49c7c113-5044-47c8-a2d3-baaf8baf4086"). InnerVolumeSpecName "kube-api-access-hrzrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:07.449186 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.449124 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:07.482340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.482321 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrzrv\" (UniqueName: \"kubernetes.io/projected/49c7c113-5044-47c8-a2d3-baaf8baf4086-kube-api-access-hrzrv\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:07.686355 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.686327 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:07.866906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.866879 5931 generic.go:356] "Generic (PLEG): container finished" podID="80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" containerID="3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83" exitCode=0 Nov 03 21:48:07.867034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.866920 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" Nov 03 21:48:07.867034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.866967 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" event={"ID":"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e","Type":"ContainerDied","Data":"3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83"} Nov 03 21:48:07.867034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.867023 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb" event={"ID":"80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e","Type":"ContainerDied","Data":"c7210d9d062579905e2b5c4042df00ebb595bfb315689d0d2d938bae819d7938"} Nov 03 21:48:07.867159 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.867044 5931 scope.go:122] "RemoveContainer" containerID="3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83" Nov 03 21:48:07.868467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.868444 5931 generic.go:356] "Generic (PLEG): container finished" podID="49c7c113-5044-47c8-a2d3-baaf8baf4086" containerID="f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae" exitCode=0 Nov 03 21:48:07.868560 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.868511 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" event={"ID":"49c7c113-5044-47c8-a2d3-baaf8baf4086","Type":"ContainerDied","Data":"f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae"} Nov 03 21:48:07.868602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.868549 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" event={"ID":"49c7c113-5044-47c8-a2d3-baaf8baf4086","Type":"ContainerDied","Data":"50e0bbf4a3f3cfeffb4dfce61b372885b8db69d1311a86fff210a2d4cd4adef3"} Nov 03 21:48:07.868690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.868656 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788" Nov 03 21:48:07.878768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.878706 5931 scope.go:122] "RemoveContainer" containerID="3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83" Nov 03 21:48:07.879060 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:07.879036 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83\": container with ID starting with 3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83 not found: ID does not exist" containerID="3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83" Nov 03 21:48:07.879141 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.879071 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83"} err="failed to get container status \"3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83\": rpc error: code = NotFound desc = could not find container \"3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83\": container with ID starting with 3c007965386857583c6fa899e56648e63f72556d273762c3d2ef93ed43f02d83 not found: ID does not exist" Nov 03 21:48:07.879141 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.879093 5931 scope.go:122] "RemoveContainer" containerID="f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae" Nov 03 21:48:07.888387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.888227 5931 scope.go:122] "RemoveContainer" containerID="f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae" Nov 03 21:48:07.888650 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:07.888489 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae\": container with ID starting with f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae not found: ID does not exist" containerID="f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae" Nov 03 21:48:07.888650 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.888509 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae"} err="failed to get container status \"f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae\": rpc error: code = NotFound desc = could not find container \"f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae\": container with ID starting with f0f0485f69f1b34e257916fa42df36ffe7df3685262543fb79258b971cf569ae not found: ID does not exist" Nov 03 21:48:07.904431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:07.904408 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:08.030563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:08.030536 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb"] Nov 03 21:48:08.079348 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:08.079326 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-8b7dc98d4-xplcb"] Nov 03 21:48:08.141224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:08.141171 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788"] Nov 03 21:48:08.146305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:08.146285 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-6c9f778c47-xk788"] Nov 03 21:48:08.508549 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:08.508479 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c7c113-5044-47c8-a2d3-baaf8baf4086" path="/var/lib/kubelet/pods/49c7c113-5044-47c8-a2d3-baaf8baf4086/volumes" Nov 03 21:48:08.509106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:08.509081 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" path="/var/lib/kubelet/pods/80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e/volumes" Nov 03 21:48:10.770148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:10.770115 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:10.988242 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:10.988203 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:11.201142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:11.201112 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:14.257956 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:14.257920 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8"] Nov 03 21:48:14.367888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:14.367858 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn"] Nov 03 21:48:14.715055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:14.715019 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:14.947776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:14.947740 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:16.257991 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:16.257961 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:16.470749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:16.470719 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:19.259038 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.258987 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" podUID="822f3025-5d9e-460f-bf25-e2503a851e28" containerName="payload-processing" containerID="cri-o://51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27" gracePeriod=25 Nov 03 21:48:19.368998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.368965 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" podUID="183b89a1-6abe-49eb-923f-39c17a775ecb" containerName="payload-pre-processing" containerID="cri-o://eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623" gracePeriod=25 Nov 03 21:48:19.522575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.522512 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:48:19.610572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.610542 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822f3025-5d9e-460f-bf25-e2503a851e28-kube-api-access-g7rpc\" (UniqueName: \"kubernetes.io/projected/822f3025-5d9e-460f-bf25-e2503a851e28-kube-api-access-g7rpc\") pod \"822f3025-5d9e-460f-bf25-e2503a851e28\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " Nov 03 21:48:19.610757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.610592 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822f3025-5d9e-460f-bf25-e2503a851e28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822f3025-5d9e-460f-bf25-e2503a851e28-plugins-config-volume\") pod \"822f3025-5d9e-460f-bf25-e2503a851e28\" (UID: \"822f3025-5d9e-460f-bf25-e2503a851e28\") " Nov 03 21:48:19.611048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.611021 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822f3025-5d9e-460f-bf25-e2503a851e28-plugins-config-volume" pod "822f3025-5d9e-460f-bf25-e2503a851e28" (UID: "822f3025-5d9e-460f-bf25-e2503a851e28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:19.611347 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.611327 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822f3025-5d9e-460f-bf25-e2503a851e28-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:19.613089 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.613061 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822f3025-5d9e-460f-bf25-e2503a851e28-kube-api-access-g7rpc" pod "822f3025-5d9e-460f-bf25-e2503a851e28" (UID: "822f3025-5d9e-460f-bf25-e2503a851e28"). InnerVolumeSpecName "kube-api-access-g7rpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:19.634164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.634135 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:48:19.711703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.711676 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183b89a1-6abe-49eb-923f-39c17a775ecb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183b89a1-6abe-49eb-923f-39c17a775ecb-plugins-config-volume\") pod \"183b89a1-6abe-49eb-923f-39c17a775ecb\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " Nov 03 21:48:19.711865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.711724 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183b89a1-6abe-49eb-923f-39c17a775ecb-kube-api-access-lhndx\" (UniqueName: \"kubernetes.io/projected/183b89a1-6abe-49eb-923f-39c17a775ecb-kube-api-access-lhndx\") pod \"183b89a1-6abe-49eb-923f-39c17a775ecb\" (UID: \"183b89a1-6abe-49eb-923f-39c17a775ecb\") " Nov 03 21:48:19.712129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.712104 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183b89a1-6abe-49eb-923f-39c17a775ecb-plugins-config-volume" pod "183b89a1-6abe-49eb-923f-39c17a775ecb" (UID: "183b89a1-6abe-49eb-923f-39c17a775ecb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:19.712324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.712306 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183b89a1-6abe-49eb-923f-39c17a775ecb-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:19.712373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.712331 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7rpc\" (UniqueName: \"kubernetes.io/projected/822f3025-5d9e-460f-bf25-e2503a851e28-kube-api-access-g7rpc\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:19.713943 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.713923 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183b89a1-6abe-49eb-923f-39c17a775ecb-kube-api-access-lhndx" pod "183b89a1-6abe-49eb-923f-39c17a775ecb" (UID: "183b89a1-6abe-49eb-923f-39c17a775ecb"). InnerVolumeSpecName "kube-api-access-lhndx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:19.813480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.813411 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhndx\" (UniqueName: \"kubernetes.io/projected/183b89a1-6abe-49eb-923f-39c17a775ecb-kube-api-access-lhndx\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:19.953954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.953922 5931 generic.go:356] "Generic (PLEG): container finished" podID="183b89a1-6abe-49eb-923f-39c17a775ecb" containerID="eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623" exitCode=0 Nov 03 21:48:19.954118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.953966 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" Nov 03 21:48:19.954118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.954002 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" event={"ID":"183b89a1-6abe-49eb-923f-39c17a775ecb","Type":"ContainerDied","Data":"eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623"} Nov 03 21:48:19.954118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.954064 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn" event={"ID":"183b89a1-6abe-49eb-923f-39c17a775ecb","Type":"ContainerDied","Data":"e97be600f7184d45f92cc8efac6e054df991620ea1e349abcb4ddbdb58e4b191"} Nov 03 21:48:19.954118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.954098 5931 scope.go:122] "RemoveContainer" containerID="eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623" Nov 03 21:48:19.955745 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.955721 5931 generic.go:356] "Generic (PLEG): container finished" podID="822f3025-5d9e-460f-bf25-e2503a851e28" containerID="51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27" exitCode=0 Nov 03 21:48:19.955869 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.955764 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" Nov 03 21:48:19.955869 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.955832 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" event={"ID":"822f3025-5d9e-460f-bf25-e2503a851e28","Type":"ContainerDied","Data":"51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27"} Nov 03 21:48:19.955983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.955867 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8" event={"ID":"822f3025-5d9e-460f-bf25-e2503a851e28","Type":"ContainerDied","Data":"8c0ab425686c0c0a77d8a8100dc1ba560ea92eea59829fb820d8992031cd0275"} Nov 03 21:48:19.965668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.965649 5931 scope.go:122] "RemoveContainer" containerID="eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623" Nov 03 21:48:19.965983 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:19.965965 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623\": container with ID starting with eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623 not found: ID does not exist" containerID="eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623" Nov 03 21:48:19.966048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.965996 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623"} err="failed to get container status \"eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623\": rpc error: code = NotFound desc = could not find container \"eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623\": container with ID starting with eeb49878fce58f480db8999724941ea7d790ba9ee3d589ea081712d229242623 not found: ID does not exist" Nov 03 21:48:19.966048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.966017 5931 scope.go:122] "RemoveContainer" containerID="51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27" Nov 03 21:48:19.975080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.975062 5931 scope.go:122] "RemoveContainer" containerID="51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27" Nov 03 21:48:19.975339 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:19.975310 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27\": container with ID starting with 51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27 not found: ID does not exist" containerID="51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27" Nov 03 21:48:19.975412 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:19.975340 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27"} err="failed to get container status \"51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27\": rpc error: code = NotFound desc = could not find container \"51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27\": container with ID starting with 51a0791a0fd63327be81b13d0a2bf5066ef2295c9cda55ff9ba176f72c26fc27 not found: ID does not exist" Nov 03 21:48:20.119555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:20.119522 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8"] Nov 03 21:48:20.122361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:20.122340 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1931b-5768b56976-sqxl8"] Nov 03 21:48:20.231165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:20.231138 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn"] Nov 03 21:48:20.236567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:20.236540 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1931b-7db569767-2wgtn"] Nov 03 21:48:20.506996 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:20.506914 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183b89a1-6abe-49eb-923f-39c17a775ecb" path="/var/lib/kubelet/pods/183b89a1-6abe-49eb-923f-39c17a775ecb/volumes" Nov 03 21:48:20.507497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:20.507477 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822f3025-5d9e-460f-bf25-e2503a851e28" path="/var/lib/kubelet/pods/822f3025-5d9e-460f-bf25-e2503a851e28/volumes" Nov 03 21:48:21.179210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:21.179180 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:24.663728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:24.663697 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:25.145327 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:25.145294 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:26.134663 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.134629 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj"] Nov 03 21:48:26.135392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135371 5931 cpu_manager.go:414] "containerMap: removing container" podUID="822f3025-5d9e-460f-bf25-e2503a851e28" containerName="payload-processing" Nov 03 21:48:26.135496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135394 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822f3025-5d9e-460f-bf25-e2503a851e28" containerName="payload-processing" Nov 03 21:48:26.135496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135427 5931 cpu_manager.go:414] "containerMap: removing container" podUID="183b89a1-6abe-49eb-923f-39c17a775ecb" containerName="payload-pre-processing" Nov 03 21:48:26.135496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135434 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183b89a1-6abe-49eb-923f-39c17a775ecb" containerName="payload-pre-processing" Nov 03 21:48:26.135496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135453 5931 cpu_manager.go:414] "containerMap: removing container" podUID="49c7c113-5044-47c8-a2d3-baaf8baf4086" containerName="payload-pre-processing" Nov 03 21:48:26.135496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135460 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c7c113-5044-47c8-a2d3-baaf8baf4086" containerName="payload-pre-processing" Nov 03 21:48:26.135786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135498 5931 cpu_manager.go:414] "containerMap: removing container" podUID="80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" containerName="payload-processing" Nov 03 21:48:26.135786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135506 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" containerName="payload-processing" Nov 03 21:48:26.135786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135672 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="183b89a1-6abe-49eb-923f-39c17a775ecb" containerName="payload-pre-processing" Nov 03 21:48:26.135786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135691 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c7c113-5044-47c8-a2d3-baaf8baf4086" containerName="payload-pre-processing" Nov 03 21:48:26.135786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135710 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="80afdb19-0c01-4bb2-8d37-ec9e0bf20d7e" containerName="payload-processing" Nov 03 21:48:26.135786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.135725 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="822f3025-5d9e-460f-bf25-e2503a851e28" containerName="payload-processing" Nov 03 21:48:26.139236 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.139214 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.188377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.188350 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwxw\" (UniqueName: \"kubernetes.io/projected/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-kube-api-access-wgwxw\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.188548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.188528 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.194517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.194493 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj"] Nov 03 21:48:26.240329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.240307 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1c3bed\"" Nov 03 21:48:26.240456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.240348 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1c3bed-dockercfg-wgt4h\"" Nov 03 21:48:26.289212 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.289186 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.289314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.289224 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwxw\" (UniqueName: \"kubernetes.io/projected/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-kube-api-access-wgwxw\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.289996 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.289974 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.345739 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.345720 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwxw\" (UniqueName: \"kubernetes.io/projected/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-kube-api-access-wgwxw\") pod \"payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.452659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.452574 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:26.827827 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:26.827802 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj"] Nov 03 21:48:26.830576 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:48:26.830548 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b5a385_3dc3_449d_b641_7f4e3b29ecc2.slice/crio-d05b6af80d24ddbdba94c0a636d92747ee211e2afa899dc6a8ba2b351df7b0d4 WatchSource:0}: Error finding container d05b6af80d24ddbdba94c0a636d92747ee211e2afa899dc6a8ba2b351df7b0d4: Status 404 returned error can't find the container with id d05b6af80d24ddbdba94c0a636d92747ee211e2afa899dc6a8ba2b351df7b0d4 Nov 03 21:48:27.007017 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:27.006983 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" event={"ID":"24b5a385-3dc3-449d-b641-7f4e3b29ecc2","Type":"ContainerStarted","Data":"d05b6af80d24ddbdba94c0a636d92747ee211e2afa899dc6a8ba2b351df7b0d4"} Nov 03 21:48:27.724835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:27.724781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:29.028248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:29.028219 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" event={"ID":"24b5a385-3dc3-449d-b641-7f4e3b29ecc2","Type":"ContainerStarted","Data":"9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da"} Nov 03 21:48:29.028586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:29.028274 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:29.143770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:29.143728 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" podStartSLOduration=1.057735413 podStartE2EDuration="3.143715304s" totalImagesPullingTime="2.085979891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:29.140455523 +0000 UTC m=+1976.751303338" watchObservedRunningTime="2026-11-03 21:48:29.143715304 +0000 UTC m=+1976.754563149" Nov 03 21:48:32.469765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:32.469732 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:33.069005 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:33.068968 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:48:37.071205 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.071173 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5"] Nov 03 21:48:37.076970 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.076952 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.131699 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.131675 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5"] Nov 03 21:48:37.177273 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.177247 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-204763-serving-cert\"" Nov 03 21:48:37.208925 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.208899 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-metrics-certs\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.209061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.208957 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7875d02-237e-4fd3-a271-801d47d8d62b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.209134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.209055 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.209134 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.209092 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvg5\" (UniqueName: \"kubernetes.io/projected/a7875d02-237e-4fd3-a271-801d47d8d62b-kube-api-access-9dvg5\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.310969 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.310913 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-metrics-certs\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.311126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.311027 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7875d02-237e-4fd3-a271-801d47d8d62b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.311126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.311067 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.311126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.311099 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvg5\" (UniqueName: \"kubernetes.io/projected/a7875d02-237e-4fd3-a271-801d47d8d62b-kube-api-access-9dvg5\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.311624 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:37.311586 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-serving-cert: secret "maas-api-e2e-ipp-b-204763-serving-cert" not found Nov 03 21:48:37.311838 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:37.311823 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls podName:a7875d02-237e-4fd3-a271-801d47d8d62b nodeName:}" failed. No retries permitted until 2026-11-03 21:48:37.811801508 +0000 UTC m=+1985.422649317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls") pod "maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" (UID: "a7875d02-237e-4fd3-a271-801d47d8d62b") : secret "maas-api-e2e-ipp-b-204763-serving-cert" not found Nov 03 21:48:37.312511 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.312472 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7875d02-237e-4fd3-a271-801d47d8d62b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.322563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.322476 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-metrics-certs\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.348276 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.348245 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:37.372202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.372175 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvg5\" (UniqueName: \"kubernetes.io/projected/a7875d02-237e-4fd3-a271-801d47d8d62b-kube-api-access-9dvg5\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.548034 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.548005 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:37.816868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.816841 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.819232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.819213 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:37.886228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.886203 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:37.990040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:37.990016 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:38.369596 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:38.369567 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5"] Nov 03 21:48:38.370873 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:48:38.370842 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7875d02_237e_4fd3_a271_801d47d8d62b.slice/crio-1b8ebc810e4ae3e13a2fa95d3aa6fedea763e9eb61977682f89a3a378ccdd231 WatchSource:0}: Error finding container 1b8ebc810e4ae3e13a2fa95d3aa6fedea763e9eb61977682f89a3a378ccdd231: Status 404 returned error can't find the container with id 1b8ebc810e4ae3e13a2fa95d3aa6fedea763e9eb61977682f89a3a378ccdd231 Nov 03 21:48:38.566366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:38.566340 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:39.095678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:39.095643 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" event={"ID":"a7875d02-237e-4fd3-a271-801d47d8d62b","Type":"ContainerStarted","Data":"1b8ebc810e4ae3e13a2fa95d3aa6fedea763e9eb61977682f89a3a378ccdd231"} Nov 03 21:48:39.354599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:39.354505 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:39.589291 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:39.589260 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:40.036422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:40.036395 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:48:40.104339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:40.104309 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" event={"ID":"a7875d02-237e-4fd3-a271-801d47d8d62b","Type":"ContainerStarted","Data":"d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6"} Nov 03 21:48:40.104501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:40.104441 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:40.271774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:40.271729 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" podStartSLOduration=2.438472106 podStartE2EDuration="3.27171355s" totalImagesPullingTime="833.241444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:40.269672429 +0000 UTC m=+1987.880520244" watchObservedRunningTime="2026-11-03 21:48:40.27171355 +0000 UTC m=+1987.882561406" Nov 03 21:48:41.318416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:41.318381 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7"] Nov 03 21:48:41.426094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:41.425994 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l"] Nov 03 21:48:41.791989 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:41.791961 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:42.023834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:42.023804 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:42.456339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:42.456302 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:43.101254 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:43.101218 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:43.868168 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:43.868141 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:44.110149 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:44.110121 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:46.117342 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.117314 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:48:46.319659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.319597 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" podUID="d6643bb7-9eaa-4ad7-8ee2-f582d7241752" containerName="payload-processing" containerID="cri-o://9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479" gracePeriod=25 Nov 03 21:48:46.427147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.427080 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" podUID="7a943f80-2717-4d0b-8484-cc52383809ba" containerName="payload-pre-processing" containerID="cri-o://ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4" gracePeriod=25 Nov 03 21:48:46.599326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.599014 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:48:46.702136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.702113 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:48:46.716238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.716204 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-plugins-config-volume\") pod \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " Nov 03 21:48:46.716376 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.716358 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-kube-api-access-jngrs\" (UniqueName: \"kubernetes.io/projected/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-kube-api-access-jngrs\") pod \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\" (UID: \"d6643bb7-9eaa-4ad7-8ee2-f582d7241752\") " Nov 03 21:48:46.716755 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.716725 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-plugins-config-volume" pod "d6643bb7-9eaa-4ad7-8ee2-f582d7241752" (UID: "d6643bb7-9eaa-4ad7-8ee2-f582d7241752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:46.716907 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.716886 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:46.718730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.718709 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-kube-api-access-jngrs" pod "d6643bb7-9eaa-4ad7-8ee2-f582d7241752" (UID: "d6643bb7-9eaa-4ad7-8ee2-f582d7241752"). InnerVolumeSpecName "kube-api-access-jngrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:46.818497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.818465 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a943f80-2717-4d0b-8484-cc52383809ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a943f80-2717-4d0b-8484-cc52383809ba-plugins-config-volume\") pod \"7a943f80-2717-4d0b-8484-cc52383809ba\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " Nov 03 21:48:46.818659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.818524 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a943f80-2717-4d0b-8484-cc52383809ba-kube-api-access-l5g8d\" (UniqueName: \"kubernetes.io/projected/7a943f80-2717-4d0b-8484-cc52383809ba-kube-api-access-l5g8d\") pod \"7a943f80-2717-4d0b-8484-cc52383809ba\" (UID: \"7a943f80-2717-4d0b-8484-cc52383809ba\") " Nov 03 21:48:46.818902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.818874 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a943f80-2717-4d0b-8484-cc52383809ba-plugins-config-volume" pod "7a943f80-2717-4d0b-8484-cc52383809ba" (UID: "7a943f80-2717-4d0b-8484-cc52383809ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:46.819052 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.819036 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngrs\" (UniqueName: \"kubernetes.io/projected/d6643bb7-9eaa-4ad7-8ee2-f582d7241752-kube-api-access-jngrs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:46.819298 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.819053 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a943f80-2717-4d0b-8484-cc52383809ba-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:46.820525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.820505 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a943f80-2717-4d0b-8484-cc52383809ba-kube-api-access-l5g8d" pod "7a943f80-2717-4d0b-8484-cc52383809ba" (UID: "7a943f80-2717-4d0b-8484-cc52383809ba"). InnerVolumeSpecName "kube-api-access-l5g8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:46.920157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:46.920125 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5g8d\" (UniqueName: \"kubernetes.io/projected/7a943f80-2717-4d0b-8484-cc52383809ba-kube-api-access-l5g8d\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:48:47.161575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.161534 5931 generic.go:356] "Generic (PLEG): container finished" podID="d6643bb7-9eaa-4ad7-8ee2-f582d7241752" containerID="9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479" exitCode=0 Nov 03 21:48:47.161575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.161572 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" Nov 03 21:48:47.162098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.161640 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" event={"ID":"d6643bb7-9eaa-4ad7-8ee2-f582d7241752","Type":"ContainerDied","Data":"9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479"} Nov 03 21:48:47.162098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.161694 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7" event={"ID":"d6643bb7-9eaa-4ad7-8ee2-f582d7241752","Type":"ContainerDied","Data":"07c63a704190e046c455ec4a8e2dc3762ddb14d1f7d3a2b0f0d046a0d6a39150"} Nov 03 21:48:47.162098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.161715 5931 scope.go:122] "RemoveContainer" containerID="9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479" Nov 03 21:48:47.163317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.163293 5931 generic.go:356] "Generic (PLEG): container finished" podID="7a943f80-2717-4d0b-8484-cc52383809ba" containerID="ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4" exitCode=0 Nov 03 21:48:47.163317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.163308 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" Nov 03 21:48:47.163475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.163355 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" event={"ID":"7a943f80-2717-4d0b-8484-cc52383809ba","Type":"ContainerDied","Data":"ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4"} Nov 03 21:48:47.163475 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.163393 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l" event={"ID":"7a943f80-2717-4d0b-8484-cc52383809ba","Type":"ContainerDied","Data":"558039b3ddcaf8f4069347c324ed876b9d321b9c8a946aa1b78fccf969e44bd4"} Nov 03 21:48:47.171899 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.171832 5931 scope.go:122] "RemoveContainer" containerID="9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479" Nov 03 21:48:47.172183 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:47.172162 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479\": container with ID starting with 9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479 not found: ID does not exist" containerID="9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479" Nov 03 21:48:47.172250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.172190 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479"} err="failed to get container status \"9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479\": rpc error: code = NotFound desc = could not find container \"9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479\": container with ID starting with 9f68c4509f1d65fac138cd8dac13cec73e1bfdb0e5fc2a81d56c125499d56479 not found: ID does not exist" Nov 03 21:48:47.172250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.172205 5931 scope.go:122] "RemoveContainer" containerID="ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4" Nov 03 21:48:47.180833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.180819 5931 scope.go:122] "RemoveContainer" containerID="ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4" Nov 03 21:48:47.181105 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:48:47.181079 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4\": container with ID starting with ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4 not found: ID does not exist" containerID="ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4" Nov 03 21:48:47.181171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.181115 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4"} err="failed to get container status \"ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4\": rpc error: code = NotFound desc = could not find container \"ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4\": container with ID starting with ab976807af2262263ea3df428196e0c478aeca611cb0333160cfe58b09d859f4 not found: ID does not exist" Nov 03 21:48:47.324580 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.324543 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l"] Nov 03 21:48:47.373458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.373438 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-7f7566fb67-f5g6l"] Nov 03 21:48:47.438281 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.438133 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7"] Nov 03 21:48:47.441709 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:47.441685 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6bffb44c96-kf4s7"] Nov 03 21:48:48.043157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.043109 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:48.251678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.251627 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:48.288260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.288222 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd"] Nov 03 21:48:48.288928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.288907 5931 cpu_manager.go:414] "containerMap: removing container" podUID="d6643bb7-9eaa-4ad7-8ee2-f582d7241752" containerName="payload-processing" Nov 03 21:48:48.288928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.288928 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6643bb7-9eaa-4ad7-8ee2-f582d7241752" containerName="payload-processing" Nov 03 21:48:48.289110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.288965 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7a943f80-2717-4d0b-8484-cc52383809ba" containerName="payload-pre-processing" Nov 03 21:48:48.289110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.288974 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a943f80-2717-4d0b-8484-cc52383809ba" containerName="payload-pre-processing" Nov 03 21:48:48.289203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.289155 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a943f80-2717-4d0b-8484-cc52383809ba" containerName="payload-pre-processing" Nov 03 21:48:48.289203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.289180 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6643bb7-9eaa-4ad7-8ee2-f582d7241752" containerName="payload-processing" Nov 03 21:48:48.294768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.294704 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:48.348361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.348326 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ae2f6f-dockercfg-ldlgq\"" Nov 03 21:48:48.348361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.348327 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ae2f6f\"" Nov 03 21:48:48.348770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.348747 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd"] Nov 03 21:48:48.459112 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.459080 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbad5a0-d723-4461-81e6-54f7c32a2d24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:48.459242 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.459174 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2br4\" (UniqueName: \"kubernetes.io/projected/dcbad5a0-d723-4461-81e6-54f7c32a2d24-kube-api-access-g2br4\") pod \"payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:48.508687 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.508659 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a943f80-2717-4d0b-8484-cc52383809ba" path="/var/lib/kubelet/pods/7a943f80-2717-4d0b-8484-cc52383809ba/volumes" Nov 03 21:48:48.509191 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.509172 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6643bb7-9eaa-4ad7-8ee2-f582d7241752" path="/var/lib/kubelet/pods/d6643bb7-9eaa-4ad7-8ee2-f582d7241752/volumes" Nov 03 21:48:48.560258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.560204 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbad5a0-d723-4461-81e6-54f7c32a2d24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:48.560258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.560242 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2br4\" (UniqueName: \"kubernetes.io/projected/dcbad5a0-d723-4461-81e6-54f7c32a2d24-kube-api-access-g2br4\") pod \"payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:48.560940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.560923 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbad5a0-d723-4461-81e6-54f7c32a2d24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:48.617313 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.617284 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2br4\" (UniqueName: \"kubernetes.io/projected/dcbad5a0-d723-4461-81e6-54f7c32a2d24-kube-api-access-g2br4\") pod \"payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:48.910667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:48.910646 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:49.286196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:49.286166 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd"] Nov 03 21:48:49.287909 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:48:49.287878 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbad5a0_d723_4461_81e6_54f7c32a2d24.slice/crio-7852c48a3580101cd4968fbb691b5e3dd313b2de45416df6beabcaaa4a309e36 WatchSource:0}: Error finding container 7852c48a3580101cd4968fbb691b5e3dd313b2de45416df6beabcaaa4a309e36: Status 404 returned error can't find the container with id 7852c48a3580101cd4968fbb691b5e3dd313b2de45416df6beabcaaa4a309e36 Nov 03 21:48:50.186407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:50.186375 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" event={"ID":"dcbad5a0-d723-4461-81e6-54f7c32a2d24","Type":"ContainerStarted","Data":"77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786"} Nov 03 21:48:50.186691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:50.186421 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" event={"ID":"dcbad5a0-d723-4461-81e6-54f7c32a2d24","Type":"ContainerStarted","Data":"7852c48a3580101cd4968fbb691b5e3dd313b2de45416df6beabcaaa4a309e36"} Nov 03 21:48:50.186691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:50.186517 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:48:50.301549 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:50.301505 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" podStartSLOduration=2.301490752 podStartE2EDuration="2.301490752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:50.299331929 +0000 UTC m=+1997.910179755" watchObservedRunningTime="2026-11-03 21:48:50.301490752 +0000 UTC m=+1997.912338566" Nov 03 21:48:52.630022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:52.629947 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:57.667668 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:57.667622 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:58.257876 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:48:58.257840 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:01.197225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:01.197197 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:49:02.646272 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:02.646242 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:02.873503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:02.873475 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:03.068567 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:49:03.068491 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:49:06.282422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:06.282386 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:06.759048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:06.758927 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:09.450401 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:09.450367 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:14.181959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:14.181910 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:14.390124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:14.390067 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:14.617927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:14.617899 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:19.140101 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:19.140071 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:23.812099 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:23.812063 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:28.545407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:28.545376 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:33.068094 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:49:33.068057 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:49:33.413839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:33.413799 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:39.842453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:39.842415 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:43.513379 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:43.513349 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:43.989725 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:43.989695 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:46.686818 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:46.686782 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:47.349036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:47.348999 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:50.307467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.307440 5931 patch_prober.go:29] interesting pod/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": read tcp 10.134.0.2:37490->10.134.0.130:15021: read: connection reset by peer" start-of-body= Nov 03 21:49:50.307980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.307514 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": read tcp 10.134.0.2:37490->10.134.0.130:15021: read: connection reset by peer" Nov 03 21:49:50.609163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.609135 5931 generic.go:356] "Generic (PLEG): container finished" podID="11b67643-5e40-46ea-a541-8377a72de3a0" containerID="f747a8e5d524d436d6aaa126eaccc9ee2d488b3195135660bcff73e4e538289f" exitCode=0 Nov 03 21:49:50.609286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.609204 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" event={"ID":"11b67643-5e40-46ea-a541-8377a72de3a0","Type":"ContainerDied","Data":"f747a8e5d524d436d6aaa126eaccc9ee2d488b3195135660bcff73e4e538289f"} Nov 03 21:49:50.609568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.609554 5931 scope.go:122] "RemoveContainer" containerID="f747a8e5d524d436d6aaa126eaccc9ee2d488b3195135660bcff73e4e538289f" Nov 03 21:49:50.610977 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.610952 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:49:50.611035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.611019 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:49:50.611069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:50.611052 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:49:51.255240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:51.255203 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:51.618209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:51.618175 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" event={"ID":"11b67643-5e40-46ea-a541-8377a72de3a0","Type":"ContainerStarted","Data":"0f83eaee05b3d4a1d3f9a31e6f3e54be3c5e5452b95373b2ff35cbc951835d7c"} Nov 03 21:49:52.059826 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:52.059748 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:49:52.059982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:52.059855 5931 patch_prober.go:29] interesting pod/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:49:52.059982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:52.059926 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:49:53.059597 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:53.059565 5931 patch_prober.go:29] interesting pod/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:49:53.060016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:53.059656 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:49:54.060222 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:54.060191 5931 patch_prober.go:29] interesting pod/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:49:54.060741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:54.060254 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:49:55.060300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.060266 5931 patch_prober.go:29] interesting pod/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:49:55.060833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.060316 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:49:55.275717 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.275681 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg"] Nov 03 21:49:55.276185 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.276154 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" podUID="7a9552d8-1a8c-431a-8289-814223f6ec2c" containerName="maas-api" containerID="cri-o://ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef" gracePeriod=30 Nov 03 21:49:55.449119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.448952 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:55.604343 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.604320 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:49:55.676175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.675988 5931 generic.go:356] "Generic (PLEG): container finished" podID="7a9552d8-1a8c-431a-8289-814223f6ec2c" containerID="ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef" exitCode=0 Nov 03 21:49:55.676175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.676031 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" Nov 03 21:49:55.676175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.676038 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" event={"ID":"7a9552d8-1a8c-431a-8289-814223f6ec2c","Type":"ContainerDied","Data":"ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef"} Nov 03 21:49:55.676175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.676093 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg" event={"ID":"7a9552d8-1a8c-431a-8289-814223f6ec2c","Type":"ContainerDied","Data":"db01795efedd4f4225ff960f97f7153d1a3cfce14456cf9ff5845a7945bda2f8"} Nov 03 21:49:55.676175 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.676120 5931 scope.go:122] "RemoveContainer" containerID="ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef" Nov 03 21:49:55.690178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.690156 5931 scope.go:122] "RemoveContainer" containerID="ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef" Nov 03 21:49:55.690501 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:49:55.690480 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef\": container with ID starting with ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef not found: ID does not exist" containerID="ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef" Nov 03 21:49:55.690601 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.690511 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef"} err="failed to get container status \"ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef\": rpc error: code = NotFound desc = could not find container \"ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef\": container with ID starting with ff0a7397c5d4428fe39c7cdf2522f3564e7de26a89e970eb304a0d59086893ef not found: ID does not exist" Nov 03 21:49:55.703362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.703333 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9552d8-1a8c-431a-8289-814223f6ec2c-kube-api-access-twct8\" (UniqueName: \"kubernetes.io/projected/7a9552d8-1a8c-431a-8289-814223f6ec2c-kube-api-access-twct8\") pod \"7a9552d8-1a8c-431a-8289-814223f6ec2c\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " Nov 03 21:49:55.703506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.703381 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9552d8-1a8c-431a-8289-814223f6ec2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9552d8-1a8c-431a-8289-814223f6ec2c-openshift-service-ca\") pod \"7a9552d8-1a8c-431a-8289-814223f6ec2c\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " Nov 03 21:49:55.703506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.703434 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-metrics-certs\") pod \"7a9552d8-1a8c-431a-8289-814223f6ec2c\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " Nov 03 21:49:55.703506 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.703450 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls\") pod \"7a9552d8-1a8c-431a-8289-814223f6ec2c\" (UID: \"7a9552d8-1a8c-431a-8289-814223f6ec2c\") " Nov 03 21:49:55.703987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.703958 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9552d8-1a8c-431a-8289-814223f6ec2c-openshift-service-ca" pod "7a9552d8-1a8c-431a-8289-814223f6ec2c" (UID: "7a9552d8-1a8c-431a-8289-814223f6ec2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:55.706386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.706360 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-metrics-certs" pod "7a9552d8-1a8c-431a-8289-814223f6ec2c" (UID: "7a9552d8-1a8c-431a-8289-814223f6ec2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:55.706468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.706365 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9552d8-1a8c-431a-8289-814223f6ec2c-kube-api-access-twct8" pod "7a9552d8-1a8c-431a-8289-814223f6ec2c" (UID: "7a9552d8-1a8c-431a-8289-814223f6ec2c"). InnerVolumeSpecName "kube-api-access-twct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:55.706959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.706934 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls" pod "7a9552d8-1a8c-431a-8289-814223f6ec2c" (UID: "7a9552d8-1a8c-431a-8289-814223f6ec2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:55.804962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.804929 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twct8\" (UniqueName: \"kubernetes.io/projected/7a9552d8-1a8c-431a-8289-814223f6ec2c-kube-api-access-twct8\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:49:55.804962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.804956 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9552d8-1a8c-431a-8289-814223f6ec2c-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:49:55.804962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.804966 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:49:55.805186 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:55.804975 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9552d8-1a8c-431a-8289-814223f6ec2c-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:49:56.059407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.059376 5931 patch_prober.go:29] interesting pod/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 21:49:56.059567 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.059424 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 21:49:56.102921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.102891 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg"] Nov 03 21:49:56.105545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.105525 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1931b-7dc784d965-q8njg"] Nov 03 21:49:56.509460 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.509416 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9552d8-1a8c-431a-8289-814223f6ec2c" path="/var/lib/kubelet/pods/7a9552d8-1a8c-431a-8289-814223f6ec2c/volumes" Nov 03 21:49:56.905509 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.905476 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv"] Nov 03 21:49:56.906257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.906239 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7a9552d8-1a8c-431a-8289-814223f6ec2c" containerName="maas-api" Nov 03 21:49:56.906331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.906259 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9552d8-1a8c-431a-8289-814223f6ec2c" containerName="maas-api" Nov 03 21:49:56.906404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.906391 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9552d8-1a8c-431a-8289-814223f6ec2c" containerName="maas-api" Nov 03 21:49:56.909768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.909745 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:56.961465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:56.961441 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ae2f6f-serving-cert\"" Nov 03 21:49:57.014368 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.014344 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv"] Nov 03 21:49:57.017045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.017021 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5z5\" (UniqueName: \"kubernetes.io/projected/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-kube-api-access-jz5z5\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.017164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.017062 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.017164 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.017130 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.017310 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.017286 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.061806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.061770 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:57.064564 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.064538 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:49:57.064962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.064942 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:49:57.065510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.065496 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:49:57.118423 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.118397 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5z5\" (UniqueName: \"kubernetes.io/projected/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-kube-api-access-jz5z5\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.118876 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.118439 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.118876 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.118564 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.118876 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.118786 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.118992 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:49:57.118954 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-serving-cert: secret "maas-api-e2e-ipp-b-ae2f6f-serving-cert" not found Nov 03 21:49:57.119026 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:49:57.119010 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls podName:191b4e57-e4be-4a3b-aa25-9879fd8dbd4a nodeName:}" failed. No retries permitted until 2026-11-03 21:49:57.618992228 +0000 UTC m=+2065.229840021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls") pod "maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" (UID: "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a") : secret "maas-api-e2e-ipp-b-ae2f6f-serving-cert" not found Nov 03 21:49:57.119178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.119154 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.121104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.121080 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.177333 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.177278 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5z5\" (UniqueName: \"kubernetes.io/projected/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-kube-api-access-jz5z5\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.342854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.342805 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:57.578003 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.577917 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:57.625042 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.625016 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.627430 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.627407 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:57.822465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:57.822436 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:58.123432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:58.123393 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:58.202306 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:58.202284 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv"] Nov 03 21:49:58.204312 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:49:58.204285 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191b4e57_e4be_4a3b_aa25_9879fd8dbd4a.slice/crio-fa63d30a0ac79ee8d4eec3344d9ebaf2d155d2a596d590adffe98aab2caf0b28 WatchSource:0}: Error finding container fa63d30a0ac79ee8d4eec3344d9ebaf2d155d2a596d590adffe98aab2caf0b28: Status 404 returned error can't find the container with id fa63d30a0ac79ee8d4eec3344d9ebaf2d155d2a596d590adffe98aab2caf0b28 Nov 03 21:49:58.697519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:58.697483 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" event={"ID":"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a","Type":"ContainerStarted","Data":"fa63d30a0ac79ee8d4eec3344d9ebaf2d155d2a596d590adffe98aab2caf0b28"} Nov 03 21:49:59.706542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:59.706500 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" event={"ID":"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a","Type":"ContainerStarted","Data":"f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02"} Nov 03 21:49:59.707056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:59.706572 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:49:59.822178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:49:59.822127 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" podStartSLOduration=2.962920201 podStartE2EDuration="3.822112589s" totalImagesPullingTime="859.192388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:59.819560889 +0000 UTC m=+2067.430408703" watchObservedRunningTime="2026-11-03 21:49:59.822112589 +0000 UTC m=+2067.432960404" Nov 03 21:50:00.205460 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:00.205422 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh"] Nov 03 21:50:00.316960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:00.316927 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt"] Nov 03 21:50:00.673026 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:00.672989 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:00.895930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:00.895896 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:01.782429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:01.782397 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-chklw_20bf40f5-8b81-478e-9e5b-c0fa554aad05/payload-processing/0.log" Nov 03 21:50:02.055169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:02.055075 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-chklw_20bf40f5-8b81-478e-9e5b-c0fa554aad05/payload-processing/0.log" Nov 03 21:50:02.752055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:02.752026 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:03.070604 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:03.070530 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:50:04.170880 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:04.170852 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:04.511891 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:04.511811 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:04.522495 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:04.522470 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h"] Nov 03 21:50:04.522746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:04.522717 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" containerID="cri-o://8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739" gracePeriod=30 Nov 03 21:50:04.783297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:04.783217 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:05.116221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.116193 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47"] Nov 03 21:50:05.120591 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.120562 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.173392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.173355 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1c3bed\"/\"default-dockercfg-xgcz5\"" Nov 03 21:50:05.173794 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.173456 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1c3bed\"/\"ipp-route-1c3bed-kserve-self-signed-certs\"" Nov 03 21:50:05.175181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.175156 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47"] Nov 03 21:50:05.206574 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.206545 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" podUID="cce2c9e8-6f22-4bbd-a08b-652366f03835" containerName="payload-processing" containerID="cri-o://747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954" gracePeriod=25 Nov 03 21:50:05.220287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.220266 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1c3bed\"/\"openshift-service-ca.crt\"" Nov 03 21:50:05.220375 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.220318 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1c3bed\"/\"kube-root-ca.crt\"" Nov 03 21:50:05.223000 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.222979 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm69j\" (UniqueName: \"kubernetes.io/projected/aea49457-8cd1-4b84-a6f2-708ba489ad06-kube-api-access-jm69j\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.223091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.223019 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-tmp-dir\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.223226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.223202 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-home\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.223286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.223233 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-model-cache\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.223355 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.223327 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-dshm\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.223476 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.223453 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aea49457-8cd1-4b84-a6f2-708ba489ad06-tls-certs\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.274535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.274508 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:05.317476 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.317438 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" podUID="c738b8a9-c20a-465a-b5bd-09bcf1d647fd" containerName="payload-pre-processing" containerID="cri-o://e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de" gracePeriod=25 Nov 03 21:50:05.325105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.325080 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-home\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.325198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.325128 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-model-cache\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.325252 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.325200 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-dshm\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.325305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.325267 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aea49457-8cd1-4b84-a6f2-708ba489ad06-tls-certs\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.325353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.325335 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm69j\" (UniqueName: \"kubernetes.io/projected/aea49457-8cd1-4b84-a6f2-708ba489ad06-kube-api-access-jm69j\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.325408 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.325375 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-tmp-dir\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.326106 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.326083 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-model-cache\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.326211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.326139 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-home\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.326285 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.326262 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-tmp-dir\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.328161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.328134 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-dshm\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.328860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.328840 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aea49457-8cd1-4b84-a6f2-708ba489ad06-tls-certs\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.389092 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.389057 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm69j\" (UniqueName: \"kubernetes.io/projected/aea49457-8cd1-4b84-a6f2-708ba489ad06-kube-api-access-jm69j\") pod \"ipp-route-1c3bed-kserve-777c5f6d9c-cvs47\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.435728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.435555 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:05.486860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.486493 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:50:05.629126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.629089 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce2c9e8-6f22-4bbd-a08b-652366f03835-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce2c9e8-6f22-4bbd-a08b-652366f03835-plugins-config-volume\") pod \"cce2c9e8-6f22-4bbd-a08b-652366f03835\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " Nov 03 21:50:05.629297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.629283 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce2c9e8-6f22-4bbd-a08b-652366f03835-kube-api-access-kdhgx\" (UniqueName: \"kubernetes.io/projected/cce2c9e8-6f22-4bbd-a08b-652366f03835-kube-api-access-kdhgx\") pod \"cce2c9e8-6f22-4bbd-a08b-652366f03835\" (UID: \"cce2c9e8-6f22-4bbd-a08b-652366f03835\") " Nov 03 21:50:05.629525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.629498 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce2c9e8-6f22-4bbd-a08b-652366f03835-plugins-config-volume" pod "cce2c9e8-6f22-4bbd-a08b-652366f03835" (UID: "cce2c9e8-6f22-4bbd-a08b-652366f03835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:50:05.629894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.629868 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce2c9e8-6f22-4bbd-a08b-652366f03835-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:05.631962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.631939 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce2c9e8-6f22-4bbd-a08b-652366f03835-kube-api-access-kdhgx" pod "cce2c9e8-6f22-4bbd-a08b-652366f03835" (UID: "cce2c9e8-6f22-4bbd-a08b-652366f03835"). InnerVolumeSpecName "kube-api-access-kdhgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:05.644158 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.644137 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:50:05.718483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.718462 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:50:05.730280 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.730261 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-kube-api-access-d6vwv\" (UniqueName: \"kubernetes.io/projected/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-kube-api-access-d6vwv\") pod \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " Nov 03 21:50:05.730464 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.730446 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-plugins-config-volume\") pod \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\" (UID: \"c738b8a9-c20a-465a-b5bd-09bcf1d647fd\") " Nov 03 21:50:05.730798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.730777 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-plugins-config-volume" pod "c738b8a9-c20a-465a-b5bd-09bcf1d647fd" (UID: "c738b8a9-c20a-465a-b5bd-09bcf1d647fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:50:05.730878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.730797 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdhgx\" (UniqueName: \"kubernetes.io/projected/cce2c9e8-6f22-4bbd-a08b-652366f03835-kube-api-access-kdhgx\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:05.732357 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.732340 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-kube-api-access-d6vwv" pod "c738b8a9-c20a-465a-b5bd-09bcf1d647fd" (UID: "c738b8a9-c20a-465a-b5bd-09bcf1d647fd"). InnerVolumeSpecName "kube-api-access-d6vwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:05.764135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.764108 5931 generic.go:356] "Generic (PLEG): container finished" podID="c738b8a9-c20a-465a-b5bd-09bcf1d647fd" containerID="e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de" exitCode=0 Nov 03 21:50:05.764261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.764146 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" Nov 03 21:50:05.764261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.764204 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" event={"ID":"c738b8a9-c20a-465a-b5bd-09bcf1d647fd","Type":"ContainerDied","Data":"e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de"} Nov 03 21:50:05.764359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.764267 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt" event={"ID":"c738b8a9-c20a-465a-b5bd-09bcf1d647fd","Type":"ContainerDied","Data":"a1e918170703d35870794848032f4b4eb3d3a8d574bb89d30743817591c97449"} Nov 03 21:50:05.764359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.764297 5931 scope.go:122] "RemoveContainer" containerID="e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de" Nov 03 21:50:05.765922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.765897 5931 generic.go:356] "Generic (PLEG): container finished" podID="cce2c9e8-6f22-4bbd-a08b-652366f03835" containerID="747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954" exitCode=0 Nov 03 21:50:05.766041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.765935 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" Nov 03 21:50:05.766041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.765980 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" event={"ID":"cce2c9e8-6f22-4bbd-a08b-652366f03835","Type":"ContainerDied","Data":"747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954"} Nov 03 21:50:05.766041 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.766028 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh" event={"ID":"cce2c9e8-6f22-4bbd-a08b-652366f03835","Type":"ContainerDied","Data":"c6f6a623524ef40578b4717152ec9d478a85202ff098fbe65ec69a45b06d3f32"} Nov 03 21:50:05.780330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.780178 5931 scope.go:122] "RemoveContainer" containerID="e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de" Nov 03 21:50:05.780807 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:05.780546 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de\": container with ID starting with e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de not found: ID does not exist" containerID="e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de" Nov 03 21:50:05.780807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.780583 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de"} err="failed to get container status \"e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de\": rpc error: code = NotFound desc = could not find container \"e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de\": container with ID starting with e9fe6b2b647da2ebc7be0065fb2566208a28c74f09749700e10f68011ed427de not found: ID does not exist" Nov 03 21:50:05.780807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.780785 5931 scope.go:122] "RemoveContainer" containerID="747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954" Nov 03 21:50:05.792007 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.791986 5931 scope.go:122] "RemoveContainer" containerID="747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954" Nov 03 21:50:05.792269 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:05.792246 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954\": container with ID starting with 747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954 not found: ID does not exist" containerID="747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954" Nov 03 21:50:05.792351 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.792274 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954"} err="failed to get container status \"747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954\": rpc error: code = NotFound desc = could not find container \"747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954\": container with ID starting with 747556d06f7fe4e470d2701570c37c7ab3b0f9867cab591c362cfe8f7fffa954 not found: ID does not exist" Nov 03 21:50:05.831972 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.831947 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:05.831972 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.831970 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vwv\" (UniqueName: \"kubernetes.io/projected/c738b8a9-c20a-465a-b5bd-09bcf1d647fd-kube-api-access-d6vwv\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:05.872885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.872862 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47"] Nov 03 21:50:05.874415 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:50:05.874394 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaea49457_8cd1_4b84_a6f2_708ba489ad06.slice/crio-f21ed6fc16624a7ce2d6d381a596a903025f9d938016a29f53539e93857936a3 WatchSource:0}: Error finding container f21ed6fc16624a7ce2d6d381a596a903025f9d938016a29f53539e93857936a3: Status 404 returned error can't find the container with id f21ed6fc16624a7ce2d6d381a596a903025f9d938016a29f53539e93857936a3 Nov 03 21:50:05.989417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.989385 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt"] Nov 03 21:50:05.994641 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:05.994595 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7d4d67d77f-7bmmt"] Nov 03 21:50:06.115924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:06.115893 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh"] Nov 03 21:50:06.120142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:06.120120 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-68c49b698b-9bwvh"] Nov 03 21:50:06.507338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:06.507271 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c738b8a9-c20a-465a-b5bd-09bcf1d647fd" path="/var/lib/kubelet/pods/c738b8a9-c20a-465a-b5bd-09bcf1d647fd/volumes" Nov 03 21:50:06.507708 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:06.507677 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce2c9e8-6f22-4bbd-a08b-652366f03835" path="/var/lib/kubelet/pods/cce2c9e8-6f22-4bbd-a08b-652366f03835/volumes" Nov 03 21:50:06.773497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:06.773428 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" event={"ID":"aea49457-8cd1-4b84-a6f2-708ba489ad06","Type":"ContainerStarted","Data":"40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e"} Nov 03 21:50:06.773497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:06.773472 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" event={"ID":"aea49457-8cd1-4b84-a6f2-708ba489ad06","Type":"ContainerStarted","Data":"f21ed6fc16624a7ce2d6d381a596a903025f9d938016a29f53539e93857936a3"} Nov 03 21:50:06.889119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:06.889076 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" podStartSLOduration=1.889061221 podStartE2EDuration="1.889061221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:06.887432238 +0000 UTC m=+2074.498280053" watchObservedRunningTime="2026-11-03 21:50:06.889061221 +0000 UTC m=+2074.499909079" Nov 03 21:50:07.435985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:07.435960 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:07.708457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:07.708368 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:09.261697 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.261659 5931 patch_prober.go:29] interesting pod/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:50:09.262063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.261716 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:50:09.781681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.781654 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:50:09.795555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.795499 5931 generic.go:356] "Generic (PLEG): container finished" podID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerID="8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739" exitCode=0 Nov 03 21:50:09.795555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.795545 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" Nov 03 21:50:09.795720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.795581 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" event={"ID":"892329ca-8914-4602-ae6a-88a3d8666cdf","Type":"ContainerDied","Data":"8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739"} Nov 03 21:50:09.795720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.795680 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h" event={"ID":"892329ca-8914-4602-ae6a-88a3d8666cdf","Type":"ContainerDied","Data":"73e9de3d9b6fcb7c80dacd2764cc37220f8fb0d6bc2316d4f26d2120a59f3b01"} Nov 03 21:50:09.795720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.795705 5931 scope.go:122] "RemoveContainer" containerID="8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739" Nov 03 21:50:09.805187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.805168 5931 scope.go:122] "RemoveContainer" containerID="a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e" Nov 03 21:50:09.814100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.814076 5931 scope.go:122] "RemoveContainer" containerID="8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739" Nov 03 21:50:09.814333 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:09.814317 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739\": container with ID starting with 8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739 not found: ID does not exist" containerID="8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739" Nov 03 21:50:09.814393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.814341 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739"} err="failed to get container status \"8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739\": rpc error: code = NotFound desc = could not find container \"8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739\": container with ID starting with 8c277cdfbef1694a82274c9aa6c017c08269de31a83bb02c9c6054f7bd9fc739 not found: ID does not exist" Nov 03 21:50:09.814393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.814359 5931 scope.go:122] "RemoveContainer" containerID="a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e" Nov 03 21:50:09.814635 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:09.814591 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e\": container with ID starting with a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e not found: ID does not exist" containerID="a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e" Nov 03 21:50:09.814683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.814629 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e"} err="failed to get container status \"a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e\": rpc error: code = NotFound desc = could not find container \"a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e\": container with ID starting with a7645e22dfb877e787d89d03100da659ac18b15cec447e22fe0ccfb84ab4666e not found: ID does not exist" Nov 03 21:50:09.871636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871601 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-kube-api-access-gtdq4\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-kube-api-access-gtdq4\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.871722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871650 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-credential-socket\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.871722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871687 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-certs\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.871722 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871720 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-data\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.871835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871820 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-podinfo\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.871883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871869 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-token\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.871938 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871925 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-socket\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.872011 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871971 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-credential-socket" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:50:09.872074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871989 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-certs" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:50:09.872074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.871981 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892329ca-8914-4602-ae6a-88a3d8666cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892329ca-8914-4602-ae6a-88a3d8666cdf-istiod-ca-cert\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.872074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872046 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-data" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:50:09.872224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872092 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-envoy\") pod \"892329ca-8914-4602-ae6a-88a3d8666cdf\" (UID: \"892329ca-8914-4602-ae6a-88a3d8666cdf\") " Nov 03 21:50:09.872224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872209 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-socket" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:50:09.872297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872283 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892329ca-8914-4602-ae6a-88a3d8666cdf-istiod-ca-cert" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:50:09.872566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872540 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.872566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872557 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892329ca-8914-4602-ae6a-88a3d8666cdf-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.872566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872566 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.872782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872575 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.872782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.872593 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.873858 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.873834 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-kube-api-access-gtdq4" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "kube-api-access-gtdq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:09.874045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.874021 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-podinfo" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:50:09.874257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.874236 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-envoy" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:50:09.874367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.874349 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-token" pod "892329ca-8914-4602-ae6a-88a3d8666cdf" (UID: "892329ca-8914-4602-ae6a-88a3d8666cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:09.973823 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.973792 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.973930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.973827 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtdq4\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-kube-api-access-gtdq4\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.973930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.973844 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:09.973930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:09.973857 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892329ca-8914-4602-ae6a-88a3d8666cdf-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:10.216295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:10.216267 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h"] Nov 03 21:50:10.219826 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:10.219799 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1931b-openshift-default-679465bf47-4w76h"] Nov 03 21:50:10.506211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:10.506151 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" path="/var/lib/kubelet/pods/892329ca-8914-4602-ae6a-88a3d8666cdf/volumes" Nov 03 21:50:12.633021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:12.632945 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:13.465587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:13.465555 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:13.689105 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:13.689075 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:15.436068 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:15.436037 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:15.448065 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:15.448043 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:15.847478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:15.847455 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:50:16.852897 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:16.852869 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf"] Nov 03 21:50:16.962584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:16.962544 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst"] Nov 03 21:50:17.316775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:17.316701 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:17.544895 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:17.544864 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:21.853277 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:21.853239 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" podUID="a09d3c35-08bf-4dfd-b115-757ec3e92bbc" containerName="payload-processing" containerID="cri-o://9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9" gracePeriod=25 Nov 03 21:50:21.962935 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:21.962906 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" podUID="7a1ffa2a-19aa-4953-94f9-e9043cf69c28" containerName="payload-pre-processing" containerID="cri-o://ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1" gracePeriod=25 Nov 03 21:50:22.124240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.124179 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:50:22.215778 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.215749 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-kube-api-access-v8gv9\" (UniqueName: \"kubernetes.io/projected/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-kube-api-access-v8gv9\") pod \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " Nov 03 21:50:22.215911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.215885 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-plugins-config-volume\") pod \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\" (UID: \"a09d3c35-08bf-4dfd-b115-757ec3e92bbc\") " Nov 03 21:50:22.216246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.216219 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-plugins-config-volume" pod "a09d3c35-08bf-4dfd-b115-757ec3e92bbc" (UID: "a09d3c35-08bf-4dfd-b115-757ec3e92bbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:50:22.216397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.216380 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:22.217810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.217793 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-kube-api-access-v8gv9" pod "a09d3c35-08bf-4dfd-b115-757ec3e92bbc" (UID: "a09d3c35-08bf-4dfd-b115-757ec3e92bbc"). InnerVolumeSpecName "kube-api-access-v8gv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:22.232232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.232212 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:50:22.317409 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.317384 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-kube-api-access-r6kzk\") pod \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " Nov 03 21:50:22.317537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.317474 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-plugins-config-volume\") pod \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\" (UID: \"7a1ffa2a-19aa-4953-94f9-e9043cf69c28\") " Nov 03 21:50:22.317840 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.317824 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8gv9\" (UniqueName: \"kubernetes.io/projected/a09d3c35-08bf-4dfd-b115-757ec3e92bbc-kube-api-access-v8gv9\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:22.317922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.317836 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-plugins-config-volume" pod "7a1ffa2a-19aa-4953-94f9-e9043cf69c28" (UID: "7a1ffa2a-19aa-4953-94f9-e9043cf69c28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:50:22.319514 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.319493 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-kube-api-access-r6kzk" pod "7a1ffa2a-19aa-4953-94f9-e9043cf69c28" (UID: "7a1ffa2a-19aa-4953-94f9-e9043cf69c28"). InnerVolumeSpecName "kube-api-access-r6kzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:22.419442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.419384 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-kube-api-access-r6kzk\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:22.419442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.419408 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1ffa2a-19aa-4953-94f9-e9043cf69c28-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:50:22.882714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.882686 5931 generic.go:356] "Generic (PLEG): container finished" podID="a09d3c35-08bf-4dfd-b115-757ec3e92bbc" containerID="9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9" exitCode=0 Nov 03 21:50:22.883058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.882764 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" Nov 03 21:50:22.883058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.882773 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" event={"ID":"a09d3c35-08bf-4dfd-b115-757ec3e92bbc","Type":"ContainerDied","Data":"9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9"} Nov 03 21:50:22.883058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.882837 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf" event={"ID":"a09d3c35-08bf-4dfd-b115-757ec3e92bbc","Type":"ContainerDied","Data":"968e4dbefc4eadc370192fd65ae4be836dd36d50f19e9d9785dcf92098fff63f"} Nov 03 21:50:22.883058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.882868 5931 scope.go:122] "RemoveContainer" containerID="9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9" Nov 03 21:50:22.884275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.884257 5931 generic.go:356] "Generic (PLEG): container finished" podID="7a1ffa2a-19aa-4953-94f9-e9043cf69c28" containerID="ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1" exitCode=0 Nov 03 21:50:22.884418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.884368 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" event={"ID":"7a1ffa2a-19aa-4953-94f9-e9043cf69c28","Type":"ContainerDied","Data":"ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1"} Nov 03 21:50:22.884418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.884403 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" event={"ID":"7a1ffa2a-19aa-4953-94f9-e9043cf69c28","Type":"ContainerDied","Data":"07648d39e02c742c6f4f2e270f806b26e978dafe45bd2abf220ec21dfcce1a8f"} Nov 03 21:50:22.884766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.884459 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst" Nov 03 21:50:22.892684 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.892666 5931 scope.go:122] "RemoveContainer" containerID="9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9" Nov 03 21:50:22.892934 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:22.892915 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9\": container with ID starting with 9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9 not found: ID does not exist" containerID="9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9" Nov 03 21:50:22.893002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.892943 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9"} err="failed to get container status \"9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9\": rpc error: code = NotFound desc = could not find container \"9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9\": container with ID starting with 9acf96d816b30fabe7bf21ef34a61df1617b474849a5775e4aee59ed88bc8dd9 not found: ID does not exist" Nov 03 21:50:22.893002 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.892959 5931 scope.go:122] "RemoveContainer" containerID="ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1" Nov 03 21:50:22.901468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.901448 5931 scope.go:122] "RemoveContainer" containerID="ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1" Nov 03 21:50:22.901848 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:22.901831 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1\": container with ID starting with ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1 not found: ID does not exist" containerID="ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1" Nov 03 21:50:22.901906 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:22.901855 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1"} err="failed to get container status \"ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1\": rpc error: code = NotFound desc = could not find container \"ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1\": container with ID starting with ea58e868ef52367b353e450ae1f4bc7367303f7d3ebdbfe82dc0b758509352b1 not found: ID does not exist" Nov 03 21:50:23.046119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:23.046096 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf"] Nov 03 21:50:23.095149 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:23.095130 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3c6597-fbdf5978-rfrrf"] Nov 03 21:50:23.204353 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:23.204249 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst"] Nov 03 21:50:23.207096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:23.207075 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c6597-5b95fbbc5-gbkst"] Nov 03 21:50:23.902874 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:23.902845 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:24.506234 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:24.506205 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1ffa2a-19aa-4953-94f9-e9043cf69c28" path="/var/lib/kubelet/pods/7a1ffa2a-19aa-4953-94f9-e9043cf69c28/volumes" Nov 03 21:50:24.506690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:24.506675 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09d3c35-08bf-4dfd-b115-757ec3e92bbc" path="/var/lib/kubelet/pods/a09d3c35-08bf-4dfd-b115-757ec3e92bbc/volumes" Nov 03 21:50:28.752963 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:28.752927 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:33.066910 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:50:33.066870 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:50:33.725971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:33.725938 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:35.564484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:35.564452 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:50:35.574933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:35.574908 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:50:35.752832 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:35.752799 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:37.424511 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:37.424472 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:37.705973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:37.705901 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:38.974285 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:38.974251 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:44.022294 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:44.022258 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:47.728366 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:47.728335 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:49.698009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:49.697975 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:56.138575 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:50:56.138539 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:02.462178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:02.462127 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:03.068756 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:51:03.068723 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:51:07.347672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:07.347639 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:09.948652 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.948584 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68"] Nov 03 21:51:09.949178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949162 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c738b8a9-c20a-465a-b5bd-09bcf1d647fd" containerName="payload-pre-processing" Nov 03 21:51:09.949227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949179 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c738b8a9-c20a-465a-b5bd-09bcf1d647fd" containerName="payload-pre-processing" Nov 03 21:51:09.949227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949197 5931 cpu_manager.go:414] "containerMap: removing container" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" Nov 03 21:51:09.949227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949202 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" Nov 03 21:51:09.949227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949217 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7a1ffa2a-19aa-4953-94f9-e9043cf69c28" containerName="payload-pre-processing" Nov 03 21:51:09.949227 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949222 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1ffa2a-19aa-4953-94f9-e9043cf69c28" containerName="payload-pre-processing" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949230 5931 cpu_manager.go:414] "containerMap: removing container" podUID="a09d3c35-08bf-4dfd-b115-757ec3e92bbc" containerName="payload-processing" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949234 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09d3c35-08bf-4dfd-b115-757ec3e92bbc" containerName="payload-processing" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949252 5931 cpu_manager.go:414] "containerMap: removing container" podUID="cce2c9e8-6f22-4bbd-a08b-652366f03835" containerName="payload-processing" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949256 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce2c9e8-6f22-4bbd-a08b-652366f03835" containerName="payload-processing" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949268 5931 cpu_manager.go:414] "containerMap: removing container" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949272 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949350 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce2c9e8-6f22-4bbd-a08b-652366f03835" containerName="payload-processing" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949360 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949367 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c738b8a9-c20a-465a-b5bd-09bcf1d647fd" containerName="payload-pre-processing" Nov 03 21:51:09.949374 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949376 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="892329ca-8914-4602-ae6a-88a3d8666cdf" containerName="istio-proxy" Nov 03 21:51:09.949698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949383 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1ffa2a-19aa-4953-94f9-e9043cf69c28" containerName="payload-pre-processing" Nov 03 21:51:09.949698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.949389 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09d3c35-08bf-4dfd-b115-757ec3e92bbc" containerName="payload-processing" Nov 03 21:51:09.952915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.952900 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:09.991137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.991107 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tmp-dir\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:09.991277 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.991247 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-dshm\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:09.991329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.991284 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-home\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:09.991329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.991308 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-model-cache\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:09.991434 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.991339 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tls-certs\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:09.991516 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:09.991498 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mfc\" (UniqueName: \"kubernetes.io/projected/0ec72e18-acc9-46dc-9d32-69bda7d11a18-kube-api-access-69mfc\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.054548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.054522 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"openshift-service-ca.crt\"" Nov 03 21:51:10.054786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.054768 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"kube-root-ca.crt\"" Nov 03 21:51:10.054983 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.054965 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"tenantcfg-model-6ddbba-kserve-self-signed-certs\"" Nov 03 21:51:10.055042 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.054963 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"default-dockercfg-x5nkn\"" Nov 03 21:51:10.059300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.059278 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68"] Nov 03 21:51:10.093455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.093299 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tmp-dir\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.094256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.093699 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-dshm\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.094256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.093757 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-home\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.094256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.093798 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-model-cache\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.094256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.093848 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tls-certs\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.094256 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.093889 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mfc\" (UniqueName: \"kubernetes.io/projected/0ec72e18-acc9-46dc-9d32-69bda7d11a18-kube-api-access-69mfc\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.096036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.095142 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tmp-dir\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.096036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.095170 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-model-cache\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.096036 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.095496 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-home\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.100128 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.100101 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-dshm\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.101835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.101809 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tls-certs\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.147974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.147933 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:10.150951 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.150928 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mfc\" (UniqueName: \"kubernetes.io/projected/0ec72e18-acc9-46dc-9d32-69bda7d11a18-kube-api-access-69mfc\") pod \"tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.264791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.264728 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:10.644910 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.644870 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68"] Nov 03 21:51:10.650205 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:51:10.650179 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ec72e18_acc9_46dc_9d32_69bda7d11a18.slice/crio-c985cabb06cb4cac1f9651a0391084a27dd11d804e61f05a20a42dd076e41a19 WatchSource:0}: Error finding container c985cabb06cb4cac1f9651a0391084a27dd11d804e61f05a20a42dd076e41a19: Status 404 returned error can't find the container with id c985cabb06cb4cac1f9651a0391084a27dd11d804e61f05a20a42dd076e41a19 Nov 03 21:51:10.880801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:10.880760 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:11.202385 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:11.202354 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" event={"ID":"0ec72e18-acc9-46dc-9d32-69bda7d11a18","Type":"ContainerStarted","Data":"efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac"} Nov 03 21:51:11.202776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:11.202399 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" event={"ID":"0ec72e18-acc9-46dc-9d32-69bda7d11a18","Type":"ContainerStarted","Data":"c985cabb06cb4cac1f9651a0391084a27dd11d804e61f05a20a42dd076e41a19"} Nov 03 21:51:11.264981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:11.264960 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:11.311268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:11.311226 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:11.316821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:11.316774 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" podStartSLOduration=2.316759273 podStartE2EDuration="2.316759273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:11.314906162 +0000 UTC m=+2138.925753976" watchObservedRunningTime="2026-11-03 21:51:11.316759273 +0000 UTC m=+2138.927607087" Nov 03 21:51:12.326987 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.326959 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7"] Nov 03 21:51:12.331463 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.331442 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:12.386897 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.386866 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7"] Nov 03 21:51:12.422518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.422477 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkcqs\" (UniqueName: \"kubernetes.io/projected/ac23ffb4-19d7-4d35-8344-8c116fb287c7-kube-api-access-fkcqs\") pod \"payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:12.422678 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.422547 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac23ffb4-19d7-4d35-8344-8c116fb287c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:12.431637 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.431594 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-204763\"" Nov 03 21:51:12.431764 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.431742 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-204763-dockercfg-c57t2\"" Nov 03 21:51:12.524103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.524082 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkcqs\" (UniqueName: \"kubernetes.io/projected/ac23ffb4-19d7-4d35-8344-8c116fb287c7-kube-api-access-fkcqs\") pod \"payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:12.524218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.524135 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac23ffb4-19d7-4d35-8344-8c116fb287c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:12.524809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.524791 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac23ffb4-19d7-4d35-8344-8c116fb287c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:12.580163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.580112 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkcqs\" (UniqueName: \"kubernetes.io/projected/ac23ffb4-19d7-4d35-8344-8c116fb287c7-kube-api-access-fkcqs\") pod \"payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:12.656478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:12.656439 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:13.041860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.041834 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7"] Nov 03 21:51:13.043511 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:51:13.043487 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac23ffb4_19d7_4d35_8344_8c116fb287c7.slice/crio-ce6a607fe524cd5941055495471414a5f286eb51430147dbd990cf42c07fb68f WatchSource:0}: Error finding container ce6a607fe524cd5941055495471414a5f286eb51430147dbd990cf42c07fb68f: Status 404 returned error can't find the container with id ce6a607fe524cd5941055495471414a5f286eb51430147dbd990cf42c07fb68f Nov 03 21:51:13.120508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.120483 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7"] Nov 03 21:51:13.124406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.124386 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.181652 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.181625 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7"] Nov 03 21:51:13.217969 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.217940 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" event={"ID":"ac23ffb4-19d7-4d35-8344-8c116fb287c7","Type":"ContainerStarted","Data":"d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430"} Nov 03 21:51:13.218113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.217993 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" event={"ID":"ac23ffb4-19d7-4d35-8344-8c116fb287c7","Type":"ContainerStarted","Data":"ce6a607fe524cd5941055495471414a5f286eb51430147dbd990cf42c07fb68f"} Nov 03 21:51:13.218113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.218026 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:13.232786 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.232763 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.232974 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.232957 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnjm\" (UniqueName: \"kubernetes.io/projected/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-kube-api-access-nxnjm\") pod \"payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.334061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.333996 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.334445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.334233 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjm\" (UniqueName: \"kubernetes.io/projected/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-kube-api-access-nxnjm\") pod \"payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.334770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.334745 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.352492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.352443 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" podStartSLOduration=1.352417872 podStartE2EDuration="1.352417872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:13.349426245 +0000 UTC m=+2140.960274062" watchObservedRunningTime="2026-11-03 21:51:13.352417872 +0000 UTC m=+2140.963265686" Nov 03 21:51:13.390425 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.390399 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnjm\" (UniqueName: \"kubernetes.io/projected/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-kube-api-access-nxnjm\") pod \"payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.438998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.438975 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:13.819130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.819104 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7"] Nov 03 21:51:13.821180 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:51:13.821154 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0910c6_14aa_4e2d_b0f1_a08099e558a0.slice/crio-62e85e0a94bb3231913e5e44faf74e132a5e6313d29a6d14d8669d726cfb743a WatchSource:0}: Error finding container 62e85e0a94bb3231913e5e44faf74e132a5e6313d29a6d14d8669d726cfb743a: Status 404 returned error can't find the container with id 62e85e0a94bb3231913e5e44faf74e132a5e6313d29a6d14d8669d726cfb743a Nov 03 21:51:13.934960 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:13.934927 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:14.228557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:14.228523 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" event={"ID":"3b0910c6-14aa-4e2d-b0f1-a08099e558a0","Type":"ContainerStarted","Data":"d7db78142ca1b7b567b50c0e8d638b8f249348974b8eea482dd83eef35d7b933"} Nov 03 21:51:14.228785 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:14.228586 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" event={"ID":"3b0910c6-14aa-4e2d-b0f1-a08099e558a0","Type":"ContainerStarted","Data":"62e85e0a94bb3231913e5e44faf74e132a5e6313d29a6d14d8669d726cfb743a"} Nov 03 21:51:14.228859 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:14.228830 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:14.346647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:14.341108 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" podStartSLOduration=1.341095687 podStartE2EDuration="1.341095687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:14.340516023 +0000 UTC m=+2141.951363851" watchObservedRunningTime="2026-11-03 21:51:14.341095687 +0000 UTC m=+2141.951943506" Nov 03 21:51:18.747690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:18.747657 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:20.265402 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:20.265373 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:20.278241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:20.278209 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:21.276096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:21.276069 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:51:22.337990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:22.337947 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm"] Nov 03 21:51:22.338631 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:22.338267 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" containerID="cri-o://8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b" gracePeriod=30 Nov 03 21:51:22.382589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:22.382558 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8"] Nov 03 21:51:22.382860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:22.382818 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" containerID="cri-o://af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41" gracePeriod=30 Nov 03 21:51:22.633012 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:22.632980 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd"] Nov 03 21:51:22.980754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:22.980675 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:23.221059 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:23.221025 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:23.853764 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:23.853735 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:24.231555 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:24.231485 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:51:25.239468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:25.239445 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:51:27.634061 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.634027 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" podUID="dcbad5a0-d723-4461-81e6-54f7c32a2d24" containerName="payload-processing" containerID="cri-o://77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786" gracePeriod=25 Nov 03 21:51:27.721630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.720391 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:51:27.749403 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.749381 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:51:27.882478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.882421 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:51:27.906861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.906823 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-token\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.906893 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-certs\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907002 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-data\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907034 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-credential-socket\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907063 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-data\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907133 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-envoy\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907142 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-certs" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.907302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907166 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/618b7afa-0ed9-4294-b034-a178742f9539-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b7afa-0ed9-4294-b034-a178742f9539-istio-podinfo\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907225 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-socket\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907265 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-certs\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907300 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618b7afa-0ed9-4294-b034-a178742f9539-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b7afa-0ed9-4294-b034-a178742f9539-istiod-ca-cert\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907336 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-kube-api-access-dpghw\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-kube-api-access-dpghw\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907385 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-data" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.907563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907402 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-socket\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907436 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-credential-socket\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907494 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-istio-token\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-istio-token\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907563 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907531 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-kube-api-access-dxxkz\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-kube-api-access-dxxkz\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907624 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-podinfo\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907708 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istiod-ca-cert\") pod \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\" (UID: \"f9cc75cb-312c-46d9-9926-72ccfd720f3c\") " Nov 03 21:51:27.907893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907757 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-envoy\") pod \"618b7afa-0ed9-4294-b034-a178742f9539\" (UID: \"618b7afa-0ed9-4294-b034-a178742f9539\") " Nov 03 21:51:27.907893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.907883 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-credential-socket" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.908526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.908144 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-socket" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.908526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.908408 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:27.908526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.908430 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:27.908526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.908445 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:27.908526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.908462 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:27.908836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.908682 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-socket" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.909114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.909092 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-data" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.909848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.909524 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618b7afa-0ed9-4294-b034-a178742f9539-istiod-ca-cert" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:27.909848 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.909598 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-token" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:27.910005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.909969 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-envoy" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.910427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.910178 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-certs" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.910427 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.910296 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istiod-ca-cert" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:27.910662 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.910630 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-credential-socket" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.911832 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.911798 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-kube-api-access-dxxkz" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "kube-api-access-dxxkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:27.911986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.911955 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-podinfo" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:51:27.912937 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.912912 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618b7afa-0ed9-4294-b034-a178742f9539-istio-podinfo" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:51:27.913990 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.913746 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-envoy" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:27.914202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.914176 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-kube-api-access-dpghw" pod "f9cc75cb-312c-46d9-9926-72ccfd720f3c" (UID: "f9cc75cb-312c-46d9-9926-72ccfd720f3c"). InnerVolumeSpecName "kube-api-access-dpghw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:27.914293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:27.914260 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-istio-token" pod "618b7afa-0ed9-4294-b034-a178742f9539" (UID: "618b7afa-0ed9-4294-b034-a178742f9539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:28.008860 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.008834 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbad5a0-d723-4461-81e6-54f7c32a2d24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbad5a0-d723-4461-81e6-54f7c32a2d24-plugins-config-volume\") pod \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " Nov 03 21:51:28.009023 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.008875 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbad5a0-d723-4461-81e6-54f7c32a2d24-kube-api-access-g2br4\" (UniqueName: \"kubernetes.io/projected/dcbad5a0-d723-4461-81e6-54f7c32a2d24-kube-api-access-g2br4\") pod \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\" (UID: \"dcbad5a0-d723-4461-81e6-54f7c32a2d24\") " Nov 03 21:51:28.009235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009209 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbad5a0-d723-4461-81e6-54f7c32a2d24-plugins-config-volume" pod "dcbad5a0-d723-4461-81e6-54f7c32a2d24" (UID: "dcbad5a0-d723-4461-81e6-54f7c32a2d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:28.009511 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009489 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009514 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxxkz\" (UniqueName: \"kubernetes.io/projected/618b7afa-0ed9-4294-b034-a178742f9539-kube-api-access-dxxkz\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009528 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009541 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009553 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009565 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbad5a0-d723-4461-81e6-54f7c32a2d24-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009577 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009589 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618b7afa-0ed9-4294-b034-a178742f9539-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009601 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.009627 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009629 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618b7afa-0ed9-4294-b034-a178742f9539-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.010079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009642 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.010079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009654 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618b7afa-0ed9-4294-b034-a178742f9539-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.010079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009667 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpghw\" (UniqueName: \"kubernetes.io/projected/f9cc75cb-312c-46d9-9926-72ccfd720f3c-kube-api-access-dpghw\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.010079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009682 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.010079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.009693 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cc75cb-312c-46d9-9926-72ccfd720f3c-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.011670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.011644 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbad5a0-d723-4461-81e6-54f7c32a2d24-kube-api-access-g2br4" pod "dcbad5a0-d723-4461-81e6-54f7c32a2d24" (UID: "dcbad5a0-d723-4461-81e6-54f7c32a2d24"). InnerVolumeSpecName "kube-api-access-g2br4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:28.114154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.112021 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2br4\" (UniqueName: \"kubernetes.io/projected/dcbad5a0-d723-4461-81e6-54f7c32a2d24-kube-api-access-g2br4\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:28.326894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.326816 5931 generic.go:356] "Generic (PLEG): container finished" podID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerID="8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b" exitCode=0 Nov 03 21:51:28.326894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.326859 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" Nov 03 21:51:28.327117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.326900 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" event={"ID":"f9cc75cb-312c-46d9-9926-72ccfd720f3c","Type":"ContainerDied","Data":"8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b"} Nov 03 21:51:28.327117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.326955 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm" event={"ID":"f9cc75cb-312c-46d9-9926-72ccfd720f3c","Type":"ContainerDied","Data":"9480e271cd133fd1095f014caf72a9da6ad4dea7d4d80048191fa24523af989a"} Nov 03 21:51:28.327117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.326979 5931 scope.go:122] "RemoveContainer" containerID="8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b" Nov 03 21:51:28.328446 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.328417 5931 generic.go:356] "Generic (PLEG): container finished" podID="dcbad5a0-d723-4461-81e6-54f7c32a2d24" containerID="77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786" exitCode=0 Nov 03 21:51:28.328534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.328464 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" Nov 03 21:51:28.328534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.328467 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" event={"ID":"dcbad5a0-d723-4461-81e6-54f7c32a2d24","Type":"ContainerDied","Data":"77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786"} Nov 03 21:51:28.328534 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.328526 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd" event={"ID":"dcbad5a0-d723-4461-81e6-54f7c32a2d24","Type":"ContainerDied","Data":"7852c48a3580101cd4968fbb691b5e3dd313b2de45416df6beabcaaa4a309e36"} Nov 03 21:51:28.330696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.330536 5931 generic.go:356] "Generic (PLEG): container finished" podID="618b7afa-0ed9-4294-b034-a178742f9539" containerID="af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41" exitCode=0 Nov 03 21:51:28.330696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.330573 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" event={"ID":"618b7afa-0ed9-4294-b034-a178742f9539","Type":"ContainerDied","Data":"af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41"} Nov 03 21:51:28.330696 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.330656 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" event={"ID":"618b7afa-0ed9-4294-b034-a178742f9539","Type":"ContainerDied","Data":"eb2b3cec6ed2d617bcd6fa0088eb3b50bd53f46536dcc5b6acc07f22acd870a0"} Nov 03 21:51:28.330929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.330696 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8" Nov 03 21:51:28.340114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.340095 5931 scope.go:122] "RemoveContainer" containerID="8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b" Nov 03 21:51:28.340406 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:51:28.340381 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b\": container with ID starting with 8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b not found: ID does not exist" containerID="8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b" Nov 03 21:51:28.340459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.340420 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b"} err="failed to get container status \"8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b\": rpc error: code = NotFound desc = could not find container \"8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b\": container with ID starting with 8c1f8ec27bd2af846f948b8427ee8eaa1047fd519c20985a6b4321172374ec0b not found: ID does not exist" Nov 03 21:51:28.340459 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.340441 5931 scope.go:122] "RemoveContainer" containerID="77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786" Nov 03 21:51:28.349245 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.349225 5931 scope.go:122] "RemoveContainer" containerID="77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786" Nov 03 21:51:28.349467 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:51:28.349449 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786\": container with ID starting with 77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786 not found: ID does not exist" containerID="77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786" Nov 03 21:51:28.349511 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.349477 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786"} err="failed to get container status \"77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786\": rpc error: code = NotFound desc = could not find container \"77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786\": container with ID starting with 77e2381ae8fa517d62419850fc802a3496b5b5f9b3b686cec83f289093291786 not found: ID does not exist" Nov 03 21:51:28.349511 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.349498 5931 scope.go:122] "RemoveContainer" containerID="af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41" Nov 03 21:51:28.357973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.357956 5931 scope.go:122] "RemoveContainer" containerID="af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41" Nov 03 21:51:28.358230 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:51:28.358215 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41\": container with ID starting with af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41 not found: ID does not exist" containerID="af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41" Nov 03 21:51:28.358305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.358237 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41"} err="failed to get container status \"af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41\": rpc error: code = NotFound desc = could not find container \"af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41\": container with ID starting with af0be3a189574edad3e1d7e4b0b9f71f37f88cd044ffffb4e0c32b39577dba41 not found: ID does not exist" Nov 03 21:51:28.509124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.509102 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm"] Nov 03 21:51:28.557997 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.557975 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-b89cm"] Nov 03 21:51:28.666444 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.666420 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8"] Nov 03 21:51:28.671968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.671949 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-bm7r8"] Nov 03 21:51:28.745123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.745097 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:28.780094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.780061 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd"] Nov 03 21:51:28.782661 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:28.782637 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-7864df855d-xdzsd"] Nov 03 21:51:30.505733 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:30.505699 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618b7afa-0ed9-4294-b034-a178742f9539" path="/var/lib/kubelet/pods/618b7afa-0ed9-4294-b034-a178742f9539/volumes" Nov 03 21:51:30.506086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:30.506062 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbad5a0-d723-4461-81e6-54f7c32a2d24" path="/var/lib/kubelet/pods/dcbad5a0-d723-4461-81e6-54f7c32a2d24/volumes" Nov 03 21:51:30.506448 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:30.506434 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" path="/var/lib/kubelet/pods/f9cc75cb-312c-46d9-9926-72ccfd720f3c/volumes" Nov 03 21:51:32.818343 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:32.818314 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:33.071127 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:51:33.071031 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:51:33.368879 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:33.368839 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:33.651719 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:33.651629 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:34.040686 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:34.040580 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:34.402470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:34.402437 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:34.678584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:34.678515 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:34.888724 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:34.888689 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk"] Nov 03 21:51:34.888966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:34.888945 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" containerID="cri-o://4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d" gracePeriod=30 Nov 03 21:51:35.522098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:35.522062 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:35.750507 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:35.750483 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:35.979529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:35.979500 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:36.699130 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:36.699102 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:39.406496 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:39.406464 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:51:39.406883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:39.406512 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:51:40.144540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.144516 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:51:40.237943 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.237890 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-kube-api-access-crpcq\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-kube-api-access-crpcq\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.237957 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-data\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238009 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-certs\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238038 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a9b7e36-937b-475b-a231-c17b97becd85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9b7e36-937b-475b-a231-c17b97becd85-istio-podinfo\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238082 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-istio-token\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-istio-token\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238124 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-envoy\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238155 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-credential-socket\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238210 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-socket\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238259 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-data" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:40.238338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238287 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9b7e36-937b-475b-a231-c17b97becd85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9b7e36-937b-475b-a231-c17b97becd85-istiod-ca-cert\") pod \"5a9b7e36-937b-475b-a231-c17b97becd85\" (UID: \"5a9b7e36-937b-475b-a231-c17b97becd85\") " Nov 03 21:51:40.238338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238314 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-certs" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:40.238548 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238478 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-socket" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:40.238645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238590 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-credential-socket" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:40.238726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238683 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9b7e36-937b-475b-a231-c17b97becd85-istiod-ca-cert" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:40.238833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238808 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.238833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238834 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.238833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238848 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.238833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238861 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9b7e36-937b-475b-a231-c17b97becd85-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.238833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.238874 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.240418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.240397 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-envoy" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:40.240512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.240497 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-istio-token" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:40.240689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.240663 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9b7e36-937b-475b-a231-c17b97becd85-istio-podinfo" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:51:40.240772 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.240698 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-kube-api-access-crpcq" pod "5a9b7e36-937b-475b-a231-c17b97becd85" (UID: "5a9b7e36-937b-475b-a231-c17b97becd85"). InnerVolumeSpecName "kube-api-access-crpcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:40.340312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.340290 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crpcq\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-kube-api-access-crpcq\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.340312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.340313 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9b7e36-937b-475b-a231-c17b97becd85-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.340472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.340323 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9b7e36-937b-475b-a231-c17b97becd85-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.340472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.340331 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9b7e36-937b-475b-a231-c17b97becd85-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:51:40.415137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.415109 5931 generic.go:356] "Generic (PLEG): container finished" podID="5a9b7e36-937b-475b-a231-c17b97becd85" containerID="4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d" exitCode=0 Nov 03 21:51:40.415478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.415159 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" Nov 03 21:51:40.415478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.415189 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" event={"ID":"5a9b7e36-937b-475b-a231-c17b97becd85","Type":"ContainerDied","Data":"4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d"} Nov 03 21:51:40.415478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.415242 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk" event={"ID":"5a9b7e36-937b-475b-a231-c17b97becd85","Type":"ContainerDied","Data":"b8517a44d58e3d9429e68a63881075d112084de74fce254531966ff5b11fc1e1"} Nov 03 21:51:40.415478 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.415269 5931 scope.go:122] "RemoveContainer" containerID="4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d" Nov 03 21:51:40.427416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.427395 5931 scope.go:122] "RemoveContainer" containerID="1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3" Nov 03 21:51:40.436222 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.436203 5931 scope.go:122] "RemoveContainer" containerID="4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d" Nov 03 21:51:40.436496 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:51:40.436478 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d\": container with ID starting with 4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d not found: ID does not exist" containerID="4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d" Nov 03 21:51:40.436543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.436507 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d"} err="failed to get container status \"4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d\": rpc error: code = NotFound desc = could not find container \"4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d\": container with ID starting with 4d9ad71ccf5a30c80faab725e5645f9360dacdf4828c22e44fb1a4b289f28f7d not found: ID does not exist" Nov 03 21:51:40.436543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.436526 5931 scope.go:122] "RemoveContainer" containerID="1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3" Nov 03 21:51:40.436822 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:51:40.436800 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3\": container with ID starting with 1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3 not found: ID does not exist" containerID="1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3" Nov 03 21:51:40.436884 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.436828 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3"} err="failed to get container status \"1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3\": rpc error: code = NotFound desc = could not find container \"1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3\": container with ID starting with 1de61ef29f111a1bf5e62b52f6c55d2f19796af503a5cada72db2767b4a3f2d3 not found: ID does not exist" Nov 03 21:51:40.577456 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.577405 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk"] Nov 03 21:51:40.582877 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:40.582853 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s96pk"] Nov 03 21:51:41.067212 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:41.067183 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:41.292550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:41.292519 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:41.536221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:41.536178 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:42.509836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:42.509805 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" path="/var/lib/kubelet/pods/5a9b7e36-937b-475b-a231-c17b97becd85/volumes" Nov 03 21:51:42.850939 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:42.850911 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:43.059670 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:43.059642 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:43.284300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:43.284227 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:44.654947 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:44.654911 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:44.996123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:44.996050 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:45.326857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:45.326779 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:45.549413 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:45.549381 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:45.843812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:45.843773 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:46.415484 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.415455 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4"] Nov 03 21:51:46.416074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416059 5931 cpu_manager.go:414] "containerMap: removing container" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" Nov 03 21:51:46.416144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416076 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" Nov 03 21:51:46.416144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416090 5931 cpu_manager.go:414] "containerMap: removing container" podUID="dcbad5a0-d723-4461-81e6-54f7c32a2d24" containerName="payload-processing" Nov 03 21:51:46.416144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416095 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbad5a0-d723-4461-81e6-54f7c32a2d24" containerName="payload-processing" Nov 03 21:51:46.416144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416114 5931 cpu_manager.go:414] "containerMap: removing container" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" Nov 03 21:51:46.416144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416121 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" Nov 03 21:51:46.416144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416144 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" Nov 03 21:51:46.416319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416149 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" Nov 03 21:51:46.416319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416253 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="618b7afa-0ed9-4294-b034-a178742f9539" containerName="istio-proxy" Nov 03 21:51:46.416319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416262 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9cc75cb-312c-46d9-9926-72ccfd720f3c" containerName="istio-proxy" Nov 03 21:51:46.416319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416272 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" Nov 03 21:51:46.416319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416278 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" Nov 03 21:51:46.416319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.416283 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbad5a0-d723-4461-81e6-54f7c32a2d24" containerName="payload-processing" Nov 03 21:51:46.421211 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.421193 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:46.477169 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.477145 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4"] Nov 03 21:51:46.518572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.518548 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4deed14-b3f2-4167-bf32-4f76defba4b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:46.518793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.518774 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssx2x\" (UniqueName: \"kubernetes.io/projected/b4deed14-b3f2-4167-bf32-4f76defba4b6-kube-api-access-ssx2x\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:46.522842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.522824 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-62cf5c\"" Nov 03 21:51:46.522932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.522877 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-62cf5c-dockercfg-nq4lb\"" Nov 03 21:51:46.619550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.619526 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4deed14-b3f2-4167-bf32-4f76defba4b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:46.619715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.619700 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssx2x\" (UniqueName: \"kubernetes.io/projected/b4deed14-b3f2-4167-bf32-4f76defba4b6-kube-api-access-ssx2x\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:46.620172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.620155 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4deed14-b3f2-4167-bf32-4f76defba4b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:46.675588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.675538 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssx2x\" (UniqueName: \"kubernetes.io/projected/b4deed14-b3f2-4167-bf32-4f76defba4b6-kube-api-access-ssx2x\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:46.737529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:46.737506 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:47.110738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.110711 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7"] Nov 03 21:51:47.111349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.111335 5931 cpu_manager.go:414] "containerMap: removing container" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" Nov 03 21:51:47.111396 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.111352 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9b7e36-937b-475b-a231-c17b97becd85" containerName="istio-proxy" Nov 03 21:51:47.115128 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:51:47.115099 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4deed14_b3f2_4167_bf32_4f76defba4b6.slice/crio-8b868b1ca479387f53f7f0650d94eb7c5f3ab5e34e9c5259efd5eaf918fc6263 WatchSource:0}: Error finding container 8b868b1ca479387f53f7f0650d94eb7c5f3ab5e34e9c5259efd5eaf918fc6263: Status 404 returned error can't find the container with id 8b868b1ca479387f53f7f0650d94eb7c5f3ab5e34e9c5259efd5eaf918fc6263 Nov 03 21:51:47.116005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.115982 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4"] Nov 03 21:51:47.116137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.116112 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.170926 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.170669 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7"] Nov 03 21:51:47.228110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.228073 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkjv\" (UniqueName: \"kubernetes.io/projected/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-kube-api-access-cqkjv\") pod \"payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.228248 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.228217 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.329049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.329021 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkjv\" (UniqueName: \"kubernetes.io/projected/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-kube-api-access-cqkjv\") pod \"payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.329161 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.329091 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.329780 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.329761 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.385917 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.385862 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkjv\" (UniqueName: \"kubernetes.io/projected/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-kube-api-access-cqkjv\") pod \"payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.466938 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.466903 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" event={"ID":"b4deed14-b3f2-4167-bf32-4f76defba4b6","Type":"ContainerStarted","Data":"8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c"} Nov 03 21:51:47.467066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.466964 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" event={"ID":"b4deed14-b3f2-4167-bf32-4f76defba4b6","Type":"ContainerStarted","Data":"8b868b1ca479387f53f7f0650d94eb7c5f3ab5e34e9c5259efd5eaf918fc6263"} Nov 03 21:51:47.467119 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.467105 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:47.475058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.475032 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:47.584985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.584927 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" podStartSLOduration=1.584900986 podStartE2EDuration="1.584900986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:47.580563333 +0000 UTC m=+2175.191411149" watchObservedRunningTime="2026-11-03 21:51:47.584900986 +0000 UTC m=+2175.195748802" Nov 03 21:51:47.866193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:47.866165 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7"] Nov 03 21:51:47.868890 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:51:47.868864 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ae7b2_e1ea_4f83_97f5_6ea5107ca7f5.slice/crio-2187e03ee282192ef6377d0b28866b1acffef6b0dcc6d96e42e84bc85b6bc011 WatchSource:0}: Error finding container 2187e03ee282192ef6377d0b28866b1acffef6b0dcc6d96e42e84bc85b6bc011: Status 404 returned error can't find the container with id 2187e03ee282192ef6377d0b28866b1acffef6b0dcc6d96e42e84bc85b6bc011 Nov 03 21:51:48.022796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:48.022750 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:48.233275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:48.233191 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:48.475266 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:48.475222 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" event={"ID":"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5","Type":"ContainerStarted","Data":"2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60"} Nov 03 21:51:48.475477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:48.475456 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:51:48.475553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:48.475502 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" event={"ID":"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5","Type":"ContainerStarted","Data":"2187e03ee282192ef6377d0b28866b1acffef6b0dcc6d96e42e84bc85b6bc011"} Nov 03 21:51:48.589381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:48.589307 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" podStartSLOduration=1.5892902150000001 podStartE2EDuration="1.589290215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:48.587812035 +0000 UTC m=+2176.198659851" watchObservedRunningTime="2026-11-03 21:51:48.589290215 +0000 UTC m=+2176.200138030" Nov 03 21:51:49.646810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:49.646778 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:49.881491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:49.881459 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:53.290258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:53.290182 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:55.133588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:55.133558 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:55.358525 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:55.358143 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:58.030250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:58.030213 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:58.477452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:58.477428 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:51:59.484289 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:51:59.484264 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:52:02.938569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:02.938525 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:03.069807 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:52:03.069773 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:52:09.444319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:09.444284 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:15.697589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:15.697536 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:20.522962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:20.522931 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:27.101432 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:27.101399 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:30.563836 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:30.563802 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:31.052940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:31.052901 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:33.067876 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:52:33.067842 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:52:33.614861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:33.614827 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:38.339422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:38.339387 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:41.585392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:41.585360 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld"] Nov 03 21:52:41.585896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:41.585530 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" containerID="cri-o://6017118e3fd81f95b606f948793844b09e48db489e02d3a03a8019f9e18151f4" gracePeriod=30 Nov 03 21:52:43.014787 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:43.014753 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:46.852231 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.852168 5931 generic.go:356] "Generic (PLEG): container finished" podID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerID="6017118e3fd81f95b606f948793844b09e48db489e02d3a03a8019f9e18151f4" exitCode=0 Nov 03 21:52:46.852522 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.852236 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" event={"ID":"b0b6fee0-8552-4713-b61e-35efb29afde6","Type":"ContainerDied","Data":"6017118e3fd81f95b606f948793844b09e48db489e02d3a03a8019f9e18151f4"} Nov 03 21:52:46.852522 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.852283 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" event={"ID":"b0b6fee0-8552-4713-b61e-35efb29afde6","Type":"ContainerDied","Data":"aab53bc0e1034233a149c09881b5e252798e83d7cb4c2e11b2d46cbaa8486c98"} Nov 03 21:52:46.852522 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.852299 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aab53bc0e1034233a149c09881b5e252798e83d7cb4c2e11b2d46cbaa8486c98" Nov 03 21:52:46.853971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.853955 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:52:46.945857 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.945836 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-envoy\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.945965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.945867 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-kube-api-access-4kf6m\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-kube-api-access-4kf6m\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.945965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.945906 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-token\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-token\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.946126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.945986 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-certs\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.946126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946038 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-data\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.946126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946075 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-credential-socket\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.946126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946120 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b6fee0-8552-4713-b61e-35efb29afde6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b6fee0-8552-4713-b61e-35efb29afde6-istiod-ca-cert\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.946331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946167 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-podinfo\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.946331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946276 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-socket\") pod \"b0b6fee0-8552-4713-b61e-35efb29afde6\" (UID: \"b0b6fee0-8552-4713-b61e-35efb29afde6\") " Nov 03 21:52:46.946331 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946312 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-certs" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:46.946497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946333 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-credential-socket" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:46.946700 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946674 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-data" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:46.946837 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946784 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-socket" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:46.946979 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946888 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b6fee0-8552-4713-b61e-35efb29afde6-istiod-ca-cert" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:52:46.947049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946979 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:46.947049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.946998 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:46.947049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.947016 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:46.947049 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.947030 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:46.948020 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.947993 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-envoy" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:46.948122 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.948095 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-token" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:46.948215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.948194 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-kube-api-access-4kf6m" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "kube-api-access-4kf6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:46.948417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:46.948400 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-podinfo" pod "b0b6fee0-8552-4713-b61e-35efb29afde6" (UID: "b0b6fee0-8552-4713-b61e-35efb29afde6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:52:47.048622 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:47.048581 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b6fee0-8552-4713-b61e-35efb29afde6-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:47.048622 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:47.048607 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:47.048782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:47.048645 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:47.048782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:47.048655 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kf6m\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-kube-api-access-4kf6m\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:47.048782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:47.048665 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b6fee0-8552-4713-b61e-35efb29afde6-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:47.724934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:47.724893 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:47.858947 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:47.858915 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld" Nov 03 21:52:48.072082 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:48.072019 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld"] Nov 03 21:52:48.075246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:48.075221 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-9kpld"] Nov 03 21:52:48.358434 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:48.358408 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:48.505588 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:48.505562 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" path="/var/lib/kubelet/pods/b0b6fee0-8552-4713-b61e-35efb29afde6/volumes" Nov 03 21:52:51.883920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:51.883889 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:52.143739 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.143664 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7"] Nov 03 21:52:52.143896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.143853 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" podUID="273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" containerName="payload-processing" containerID="cri-o://2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60" gracePeriod=30 Nov 03 21:52:52.196178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.196155 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4"] Nov 03 21:52:52.196385 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.196363 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" podUID="b4deed14-b3f2-4167-bf32-4f76defba4b6" containerName="payload-pre-processing" containerID="cri-o://8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c" gracePeriod=30 Nov 03 21:52:52.477300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.477264 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:52.512095 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.512074 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:52:52.515306 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.515288 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:52:52.602706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.602684 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4deed14-b3f2-4167-bf32-4f76defba4b6-kube-api-access-ssx2x\" (UniqueName: \"kubernetes.io/projected/b4deed14-b3f2-4167-bf32-4f76defba4b6-kube-api-access-ssx2x\") pod \"b4deed14-b3f2-4167-bf32-4f76defba4b6\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " Nov 03 21:52:52.602816 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.602717 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-kube-api-access-cqkjv\" (UniqueName: \"kubernetes.io/projected/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-kube-api-access-cqkjv\") pod \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " Nov 03 21:52:52.602816 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.602786 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-plugins-config-volume\") pod \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\" (UID: \"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5\") " Nov 03 21:52:52.602920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.602904 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4deed14-b3f2-4167-bf32-4f76defba4b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4deed14-b3f2-4167-bf32-4f76defba4b6-plugins-config-volume\") pod \"b4deed14-b3f2-4167-bf32-4f76defba4b6\" (UID: \"b4deed14-b3f2-4167-bf32-4f76defba4b6\") " Nov 03 21:52:52.603212 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.603189 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-plugins-config-volume" pod "273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" (UID: "273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:52:52.603321 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.603293 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4deed14-b3f2-4167-bf32-4f76defba4b6-plugins-config-volume" pod "b4deed14-b3f2-4167-bf32-4f76defba4b6" (UID: "b4deed14-b3f2-4167-bf32-4f76defba4b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:52:52.603697 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.603679 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:52.603748 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.603705 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4deed14-b3f2-4167-bf32-4f76defba4b6-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:52.604995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.604977 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4deed14-b3f2-4167-bf32-4f76defba4b6-kube-api-access-ssx2x" pod "b4deed14-b3f2-4167-bf32-4f76defba4b6" (UID: "b4deed14-b3f2-4167-bf32-4f76defba4b6"). InnerVolumeSpecName "kube-api-access-ssx2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:52.605070 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.605034 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-kube-api-access-cqkjv" pod "273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" (UID: "273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5"). InnerVolumeSpecName "kube-api-access-cqkjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:52.705268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.705206 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssx2x\" (UniqueName: \"kubernetes.io/projected/b4deed14-b3f2-4167-bf32-4f76defba4b6-kube-api-access-ssx2x\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:52.705268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.705230 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkjv\" (UniqueName: \"kubernetes.io/projected/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5-kube-api-access-cqkjv\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:52:52.717864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.717838 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:52.896903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.896870 5931 generic.go:356] "Generic (PLEG): container finished" podID="b4deed14-b3f2-4167-bf32-4f76defba4b6" containerID="8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c" exitCode=0 Nov 03 21:52:52.897369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.896909 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" Nov 03 21:52:52.897369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.896946 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" event={"ID":"b4deed14-b3f2-4167-bf32-4f76defba4b6","Type":"ContainerDied","Data":"8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c"} Nov 03 21:52:52.897369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.897000 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4" event={"ID":"b4deed14-b3f2-4167-bf32-4f76defba4b6","Type":"ContainerDied","Data":"8b868b1ca479387f53f7f0650d94eb7c5f3ab5e34e9c5259efd5eaf918fc6263"} Nov 03 21:52:52.897369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.897026 5931 scope.go:122] "RemoveContainer" containerID="8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c" Nov 03 21:52:52.898656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.898631 5931 generic.go:356] "Generic (PLEG): container finished" podID="273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" containerID="2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60" exitCode=0 Nov 03 21:52:52.898765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.898693 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" event={"ID":"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5","Type":"ContainerDied","Data":"2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60"} Nov 03 21:52:52.898765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.898735 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" event={"ID":"273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5","Type":"ContainerDied","Data":"2187e03ee282192ef6377d0b28866b1acffef6b0dcc6d96e42e84bc85b6bc011"} Nov 03 21:52:52.898765 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.898755 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7" Nov 03 21:52:52.908389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.908345 5931 scope.go:122] "RemoveContainer" containerID="8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c" Nov 03 21:52:52.908659 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:52:52.908635 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c\": container with ID starting with 8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c not found: ID does not exist" containerID="8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c" Nov 03 21:52:52.908743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.908671 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c"} err="failed to get container status \"8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c\": rpc error: code = NotFound desc = could not find container \"8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c\": container with ID starting with 8375f22ba4ca7441259dff724f546388a9a592d0457822b15225b2ed5757b05c not found: ID does not exist" Nov 03 21:52:52.908743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.908693 5931 scope.go:122] "RemoveContainer" containerID="2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60" Nov 03 21:52:52.919051 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.919034 5931 scope.go:122] "RemoveContainer" containerID="2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60" Nov 03 21:52:52.919327 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:52:52.919308 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60\": container with ID starting with 2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60 not found: ID does not exist" containerID="2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60" Nov 03 21:52:52.919387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:52.919336 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60"} err="failed to get container status \"2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60\": rpc error: code = NotFound desc = could not find container \"2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60\": container with ID starting with 2e06bab3f3636f0e6295cfd506581e73e08061cab74d233d5ec4f61fb0fecd60 not found: ID does not exist" Nov 03 21:52:52.957931 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:52:52.957873 5931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4deed14_b3f2_4167_bf32_4f76defba4b6.slice/crio-8b868b1ca479387f53f7f0650d94eb7c5f3ab5e34e9c5259efd5eaf918fc6263\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4deed14_b3f2_4167_bf32_4f76defba4b6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ae7b2_e1ea_4f83_97f5_6ea5107ca7f5.slice/crio-2187e03ee282192ef6377d0b28866b1acffef6b0dcc6d96e42e84bc85b6bc011\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ae7b2_e1ea_4f83_97f5_6ea5107ca7f5.slice\": RecentStats: unable to find data in memory cache]" Nov 03 21:52:53.060406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:53.060380 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7"] Nov 03 21:52:53.062825 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:53.062806 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-6459d49646-7jzl7"] Nov 03 21:52:53.168863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:53.168841 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4"] Nov 03 21:52:53.172264 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:53.172243 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-7b445d48c5-ddsc4"] Nov 03 21:52:53.578221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:53.578187 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:53.797779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:53.797736 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:54.035397 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:54.035355 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:54.506220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:54.506189 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" path="/var/lib/kubelet/pods/273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5/volumes" Nov 03 21:52:54.506622 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:54.506593 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4deed14-b3f2-4167-bf32-4f76defba4b6" path="/var/lib/kubelet/pods/b4deed14-b3f2-4167-bf32-4f76defba4b6/volumes" Nov 03 21:52:54.595869 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:54.595839 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:59.038216 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:52:59.038124 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:00.595218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:00.595170 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:00.831714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:00.831683 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:00.842516 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:00.842487 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt"] Nov 03 21:53:00.842754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:00.842733 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" containerID="cri-o://0f83eaee05b3d4a1d3f9a31e6f3e54be3c5e5452b95373b2ff35cbc951835d7c" gracePeriod=30 Nov 03 21:53:01.045843 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:01.045755 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:02.555255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:02.555222 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:03.077212 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:53:03.077175 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:53:03.204132 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:03.204093 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:03.430699 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:03.430659 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:04.142232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:04.142200 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:05.993630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:05.993584 5931 generic.go:356] "Generic (PLEG): container finished" podID="11b67643-5e40-46ea-a541-8377a72de3a0" containerID="0f83eaee05b3d4a1d3f9a31e6f3e54be3c5e5452b95373b2ff35cbc951835d7c" exitCode=0 Nov 03 21:53:05.994043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:05.993645 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" event={"ID":"11b67643-5e40-46ea-a541-8377a72de3a0","Type":"ContainerDied","Data":"0f83eaee05b3d4a1d3f9a31e6f3e54be3c5e5452b95373b2ff35cbc951835d7c"} Nov 03 21:53:05.994043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:05.993755 5931 scope.go:122] "RemoveContainer" containerID="f747a8e5d524d436d6aaa126eaccc9ee2d488b3195135660bcff73e4e538289f" Nov 03 21:53:06.104444 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.104415 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:53:06.139467 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139416 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-data\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139491 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-certs\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139510 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-envoy\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139561 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-socket\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139636 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11b67643-5e40-46ea-a541-8377a72de3a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b67643-5e40-46ea-a541-8377a72de3a0-istio-podinfo\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139692 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b67643-5e40-46ea-a541-8377a72de3a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b67643-5e40-46ea-a541-8377a72de3a0-istiod-ca-cert\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139715 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139710 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-credential-socket\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139748 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-kube-api-access-qbczv\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-kube-api-access-qbczv\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139785 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-istio-token\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-istio-token\") pod \"11b67643-5e40-46ea-a541-8377a72de3a0\" (UID: \"11b67643-5e40-46ea-a541-8377a72de3a0\") " Nov 03 21:53:06.139845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139687 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-data" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:06.139845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139789 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-certs" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:06.140033 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.139868 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-socket" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:06.140214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.140190 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.140214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.140200 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b67643-5e40-46ea-a541-8377a72de3a0-istiod-ca-cert" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:53:06.140552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.140240 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.140552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.140255 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.140552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.140260 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-credential-socket" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:06.141835 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.141818 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11b67643-5e40-46ea-a541-8377a72de3a0-istio-podinfo" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:53:06.142088 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.142053 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-istio-token" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:06.142241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.142228 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-kube-api-access-qbczv" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "kube-api-access-qbczv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:06.142319 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.142276 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-envoy" pod "11b67643-5e40-46ea-a541-8377a72de3a0" (UID: "11b67643-5e40-46ea-a541-8377a72de3a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:06.241094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.241069 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.241094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.241095 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b67643-5e40-46ea-a541-8377a72de3a0-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.241243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.241105 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b67643-5e40-46ea-a541-8377a72de3a0-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.241243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.241114 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b67643-5e40-46ea-a541-8377a72de3a0-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.241243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.241122 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbczv\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-kube-api-access-qbczv\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:06.241243 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:06.241131 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b67643-5e40-46ea-a541-8377a72de3a0-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:07.003072 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.003049 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" Nov 03 21:53:07.003508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.003051 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt" event={"ID":"11b67643-5e40-46ea-a541-8377a72de3a0","Type":"ContainerDied","Data":"eea9ace50e8324f05bd40b07a00fd0ce4441cb5caa82504e9894b9408cdda3cd"} Nov 03 21:53:07.003508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.003170 5931 scope.go:122] "RemoveContainer" containerID="0f83eaee05b3d4a1d3f9a31e6f3e54be3c5e5452b95373b2ff35cbc951835d7c" Nov 03 21:53:07.229117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.229090 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt"] Nov 03 21:53:07.278985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.278929 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-62cf5c-openshift-default-8d4d79847-wnbgt"] Nov 03 21:53:07.407792 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.407750 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:07.654015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.653989 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:07.921300 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:07.921240 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-chklw_20bf40f5-8b81-478e-9e5b-c0fa554aad05/payload-processing/0.log" Nov 03 21:53:08.184989 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:08.184931 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-chklw_20bf40f5-8b81-478e-9e5b-c0fa554aad05/payload-processing/0.log" Nov 03 21:53:08.413854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:08.413833 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7_3b0910c6-14aa-4e2d-b0f1-a08099e558a0/payload-processing/0.log" Nov 03 21:53:08.506657 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:08.506570 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" path="/var/lib/kubelet/pods/11b67643-5e40-46ea-a541-8377a72de3a0/volumes" Nov 03 21:53:08.646807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:08.646782 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7_3b0910c6-14aa-4e2d-b0f1-a08099e558a0/payload-processing/0.log" Nov 03 21:53:09.078490 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:09.078455 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:10.633318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.633290 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:10.862679 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.862647 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:10.966445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966345 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq"] Nov 03 21:53:10.966908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966895 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" Nov 03 21:53:10.966958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966910 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" Nov 03 21:53:10.966958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966921 5931 cpu_manager.go:414] "containerMap: removing container" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" Nov 03 21:53:10.966958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966926 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" Nov 03 21:53:10.966958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966934 5931 cpu_manager.go:414] "containerMap: removing container" podUID="273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" containerName="payload-processing" Nov 03 21:53:10.966958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966939 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" containerName="payload-processing" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966965 5931 cpu_manager.go:414] "containerMap: removing container" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966970 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966986 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b4deed14-b3f2-4167-bf32-4f76defba4b6" containerName="payload-pre-processing" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.966990 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4deed14-b3f2-4167-bf32-4f76defba4b6" containerName="payload-pre-processing" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.967061 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4deed14-b3f2-4167-bf32-4f76defba4b6" containerName="payload-pre-processing" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.967074 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b6fee0-8552-4713-b61e-35efb29afde6" containerName="istio-proxy" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.967079 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.967085 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ae7b2-e1ea-4f83-97f5-6ea5107ca7f5" containerName="payload-processing" Nov 03 21:53:10.967110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.967093 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b67643-5e40-46ea-a541-8377a72de3a0" containerName="istio-proxy" Nov 03 21:53:10.971875 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.971856 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:10.984736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.984714 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc1020-8706-4143-9035-8e50bce88add-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:10.984871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:10.984851 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6sx\" (UniqueName: \"kubernetes.io/projected/fddc1020-8706-4143-9035-8e50bce88add-kube-api-access-vg6sx\") pod \"payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:11.026202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.026168 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq"] Nov 03 21:53:11.073416 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.073384 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c993b8\"" Nov 03 21:53:11.073773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.073751 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c993b8-dockercfg-h6fv7\"" Nov 03 21:53:11.086438 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.086412 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc1020-8706-4143-9035-8e50bce88add-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:11.086546 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.086501 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6sx\" (UniqueName: \"kubernetes.io/projected/fddc1020-8706-4143-9035-8e50bce88add-kube-api-access-vg6sx\") pod \"payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:11.087182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.087164 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc1020-8706-4143-9035-8e50bce88add-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:11.111407 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.111381 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f"] Nov 03 21:53:11.116128 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.116110 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.142658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.142635 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6sx\" (UniqueName: \"kubernetes.io/projected/fddc1020-8706-4143-9035-8e50bce88add-kube-api-access-vg6sx\") pod \"payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:11.171045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.171018 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f"] Nov 03 21:53:11.188035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.188012 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd29j\" (UniqueName: \"kubernetes.io/projected/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-kube-api-access-wd29j\") pod \"payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.188140 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.188044 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.285558 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.285505 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:11.288799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.288779 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd29j\" (UniqueName: \"kubernetes.io/projected/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-kube-api-access-wd29j\") pod \"payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.288846 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.288806 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.289435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.289412 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.354081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.354051 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd29j\" (UniqueName: \"kubernetes.io/projected/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-kube-api-access-wd29j\") pod \"payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.430074 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.430049 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:11.659057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.659034 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq"] Nov 03 21:53:11.660806 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:53:11.660784 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddc1020_8706_4143_9035_8e50bce88add.slice/crio-0c32526d944d61da81ad28e32b6a888e0b2fdb30ac50831996eba4d8bee7c068 WatchSource:0}: Error finding container 0c32526d944d61da81ad28e32b6a888e0b2fdb30ac50831996eba4d8bee7c068: Status 404 returned error can't find the container with id 0c32526d944d61da81ad28e32b6a888e0b2fdb30ac50831996eba4d8bee7c068 Nov 03 21:53:11.804081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:11.804060 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f"] Nov 03 21:53:11.806049 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:53:11.806023 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6839d4a_43e3_4c84_b3a4_30ca5b38283d.slice/crio-a86fc84e377c209705818c2dc59eda296830b17dc2b9d154a867f1364c3ed4ae WatchSource:0}: Error finding container a86fc84e377c209705818c2dc59eda296830b17dc2b9d154a867f1364c3ed4ae: Status 404 returned error can't find the container with id a86fc84e377c209705818c2dc59eda296830b17dc2b9d154a867f1364c3ed4ae Nov 03 21:53:12.039403 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.039327 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" event={"ID":"fddc1020-8706-4143-9035-8e50bce88add","Type":"ContainerStarted","Data":"eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad"} Nov 03 21:53:12.039403 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.039377 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" event={"ID":"fddc1020-8706-4143-9035-8e50bce88add","Type":"ContainerStarted","Data":"0c32526d944d61da81ad28e32b6a888e0b2fdb30ac50831996eba4d8bee7c068"} Nov 03 21:53:12.039586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.039406 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:12.041027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.040994 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" event={"ID":"a6839d4a-43e3-4c84-b3a4-30ca5b38283d","Type":"ContainerStarted","Data":"304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8"} Nov 03 21:53:12.041121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.041033 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" event={"ID":"a6839d4a-43e3-4c84-b3a4-30ca5b38283d","Type":"ContainerStarted","Data":"a86fc84e377c209705818c2dc59eda296830b17dc2b9d154a867f1364c3ed4ae"} Nov 03 21:53:12.041121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.041055 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:12.155064 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.155012 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" podStartSLOduration=2.15499505 podStartE2EDuration="2.15499505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:12.151237117 +0000 UTC m=+2259.762084931" watchObservedRunningTime="2026-11-03 21:53:12.15499505 +0000 UTC m=+2259.765842866" Nov 03 21:53:12.266824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:12.266777 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" podStartSLOduration=1.266763677 podStartE2EDuration="1.266763677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:12.263069859 +0000 UTC m=+2259.873917674" watchObservedRunningTime="2026-11-03 21:53:12.266763677 +0000 UTC m=+2259.877611512" Nov 03 21:53:14.366152 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:14.366122 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:19.975395 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:19.975364 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf"] Nov 03 21:53:19.975797 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:19.975564 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" podUID="56cf5107-7bdd-474c-8717-6cbf5dffc680" containerName="maas-api" containerID="cri-o://8dcb0b80dffeef8613dd4877717bf0fa109c4394c0ca2559cd549ba30df268fc" gracePeriod=30 Nov 03 21:53:20.096200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.096168 5931 generic.go:356] "Generic (PLEG): container finished" podID="56cf5107-7bdd-474c-8717-6cbf5dffc680" containerID="8dcb0b80dffeef8613dd4877717bf0fa109c4394c0ca2559cd549ba30df268fc" exitCode=0 Nov 03 21:53:20.096200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.096206 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" event={"ID":"56cf5107-7bdd-474c-8717-6cbf5dffc680","Type":"ContainerDied","Data":"8dcb0b80dffeef8613dd4877717bf0fa109c4394c0ca2559cd549ba30df268fc"} Nov 03 21:53:20.158820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.157245 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:20.234382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.234335 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:53:20.281358 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.281336 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cf5107-7bdd-474c-8717-6cbf5dffc680-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf5107-7bdd-474c-8717-6cbf5dffc680-openshift-service-ca\") pod \"56cf5107-7bdd-474c-8717-6cbf5dffc680\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " Nov 03 21:53:20.281458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.281405 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls\") pod \"56cf5107-7bdd-474c-8717-6cbf5dffc680\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " Nov 03 21:53:20.281458 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.281434 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-metrics-certs\") pod \"56cf5107-7bdd-474c-8717-6cbf5dffc680\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " Nov 03 21:53:20.281574 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.281473 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cf5107-7bdd-474c-8717-6cbf5dffc680-kube-api-access-5qtmm\" (UniqueName: \"kubernetes.io/projected/56cf5107-7bdd-474c-8717-6cbf5dffc680-kube-api-access-5qtmm\") pod \"56cf5107-7bdd-474c-8717-6cbf5dffc680\" (UID: \"56cf5107-7bdd-474c-8717-6cbf5dffc680\") " Nov 03 21:53:20.281675 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.281642 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cf5107-7bdd-474c-8717-6cbf5dffc680-openshift-service-ca" pod "56cf5107-7bdd-474c-8717-6cbf5dffc680" (UID: "56cf5107-7bdd-474c-8717-6cbf5dffc680"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:53:20.281824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.281810 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf5107-7bdd-474c-8717-6cbf5dffc680-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:20.283601 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.283568 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cf5107-7bdd-474c-8717-6cbf5dffc680-kube-api-access-5qtmm" pod "56cf5107-7bdd-474c-8717-6cbf5dffc680" (UID: "56cf5107-7bdd-474c-8717-6cbf5dffc680"). InnerVolumeSpecName "kube-api-access-5qtmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:20.283981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.283962 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-metrics-certs" pod "56cf5107-7bdd-474c-8717-6cbf5dffc680" (UID: "56cf5107-7bdd-474c-8717-6cbf5dffc680"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:53:20.284027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.283993 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls" pod "56cf5107-7bdd-474c-8717-6cbf5dffc680" (UID: "56cf5107-7bdd-474c-8717-6cbf5dffc680"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:53:20.383027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.382988 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtmm\" (UniqueName: \"kubernetes.io/projected/56cf5107-7bdd-474c-8717-6cbf5dffc680-kube-api-access-5qtmm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:20.383027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.383030 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:20.383230 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.383045 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cf5107-7bdd-474c-8717-6cbf5dffc680-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:20.663194 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.663158 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:20.871443 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:20.871410 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:21.103769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:21.103739 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" Nov 03 21:53:21.104181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:21.103742 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf" event={"ID":"56cf5107-7bdd-474c-8717-6cbf5dffc680","Type":"ContainerDied","Data":"c281d9e9189edc1f3dbabedcbe298f0388719b4c8d57d1b5a229bba1ea4da5ba"} Nov 03 21:53:21.104181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:21.103895 5931 scope.go:122] "RemoveContainer" containerID="8dcb0b80dffeef8613dd4877717bf0fa109c4394c0ca2559cd549ba30df268fc" Nov 03 21:53:21.265914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:21.265849 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf"] Nov 03 21:53:21.269476 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:21.269452 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3940f8-697879785f-cjnhf"] Nov 03 21:53:21.864779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:21.864730 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:22.068959 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:22.068928 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:22.505544 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:22.505517 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56cf5107-7bdd-474c-8717-6cbf5dffc680" path="/var/lib/kubelet/pods/56cf5107-7bdd-474c-8717-6cbf5dffc680/volumes" Nov 03 21:53:22.707429 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:22.707400 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:23.049645 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:23.049600 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:53:23.049807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:23.049786 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:53:27.330812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:27.330771 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:28.461387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:28.461351 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:28.691593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:28.691559 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:28.741794 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:28.741725 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv"] Nov 03 21:53:28.741940 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:28.741921 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" containerID="cri-o://1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa" gracePeriod=30 Nov 03 21:53:28.902953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:28.902923 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:32.609138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:32.609104 5931 patch_prober.go:29] interesting pod/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:53:32.609532 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:32.609168 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:53:33.069771 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:53:33.069694 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:53:33.697984 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:33.697957 5931 scope.go:122] "RemoveContainer" containerID="6017118e3fd81f95b606f948793844b09e48db489e02d3a03a8019f9e18151f4" Nov 03 21:53:33.985306 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:33.985280 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:53:34.113813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.113776 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-credential-socket\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.113988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.113853 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-token\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-token\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.113988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.113892 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-podinfo\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.113988 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.113919 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-certs\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.114166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114004 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-envoy\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.114166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114047 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-credential-socket" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:34.114166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114050 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-kube-api-access-m7cvp\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-kube-api-access-m7cvp\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.114166 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114111 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-socket\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.114381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114218 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-data\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.114381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114245 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-certs" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:34.114381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114332 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9655b3a-b651-41f8-91e9-3ab609c61a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9655b3a-b651-41f8-91e9-3ab609c61a79-istiod-ca-cert\") pod \"a9655b3a-b651-41f8-91e9-3ab609c61a79\" (UID: \"a9655b3a-b651-41f8-91e9-3ab609c61a79\") " Nov 03 21:53:34.114535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114428 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-socket" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:34.114535 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114514 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-data" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:34.114824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114781 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9655b3a-b651-41f8-91e9-3ab609c61a79-istiod-ca-cert" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:53:34.114945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114924 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.114945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114946 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.115091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114960 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9655b3a-b651-41f8-91e9-3ab609c61a79-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.115091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114972 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.115091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.114985 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.116260 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.116233 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-podinfo" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:53:34.116500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.116481 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-kube-api-access-m7cvp" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "kube-api-access-m7cvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:34.116782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.116757 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-token" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:34.116864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.116837 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-envoy" pod "a9655b3a-b651-41f8-91e9-3ab609c61a79" (UID: "a9655b3a-b651-41f8-91e9-3ab609c61a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:53:34.123592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.123570 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:34.188665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.188634 5931 generic.go:356] "Generic (PLEG): container finished" podID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerID="1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa" exitCode=0 Nov 03 21:53:34.188779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.188753 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" Nov 03 21:53:34.188851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.188747 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" event={"ID":"a9655b3a-b651-41f8-91e9-3ab609c61a79","Type":"ContainerDied","Data":"1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa"} Nov 03 21:53:34.188888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.188879 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv" event={"ID":"a9655b3a-b651-41f8-91e9-3ab609c61a79","Type":"ContainerDied","Data":"3cab56b5185dd484262160b20c0c0dabf50575ed04bf5dd443dffc033d0169f8"} Nov 03 21:53:34.188924 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.188913 5931 scope.go:122] "RemoveContainer" containerID="1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa" Nov 03 21:53:34.199594 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.199579 5931 scope.go:122] "RemoveContainer" containerID="bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37" Nov 03 21:53:34.208291 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.208274 5931 scope.go:122] "RemoveContainer" containerID="1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa" Nov 03 21:53:34.208529 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:53:34.208509 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa\": container with ID starting with 1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa not found: ID does not exist" containerID="1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa" Nov 03 21:53:34.208572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.208538 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa"} err="failed to get container status \"1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa\": rpc error: code = NotFound desc = could not find container \"1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa\": container with ID starting with 1411edcc6627a2fb5b8c20190b3116eb32b17b428120b020172fec9fa51268aa not found: ID does not exist" Nov 03 21:53:34.208572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.208557 5931 scope.go:122] "RemoveContainer" containerID="bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37" Nov 03 21:53:34.208833 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:53:34.208798 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37\": container with ID starting with bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37 not found: ID does not exist" containerID="bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37" Nov 03 21:53:34.208833 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.208823 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37"} err="failed to get container status \"bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37\": rpc error: code = NotFound desc = could not find container \"bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37\": container with ID starting with bb2defc37716d5e012248ce75e2b8c7e2ae9f31ddc319528a877e65ee22c1b37 not found: ID does not exist" Nov 03 21:53:34.216509 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.216489 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.216509 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.216511 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.216656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.216520 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7cvp\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-kube-api-access-m7cvp\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.216656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.216531 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9655b3a-b651-41f8-91e9-3ab609c61a79-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:53:34.365188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.365153 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv"] Nov 03 21:53:34.368146 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.368125 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3940f8-openshift-default-b9b79c699-mrlfv"] Nov 03 21:53:34.505474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:34.505421 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" path="/var/lib/kubelet/pods/a9655b3a-b651-41f8-91e9-3ab609c61a79/volumes" Nov 03 21:53:40.567851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:40.567822 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:45.208029 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:45.207985 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:48.719091 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:48.719039 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:49.965758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:49.965717 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:54.935538 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:54.935504 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:59.902771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:53:59.902729 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:03.067915 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:54:03.067881 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:54:04.600936 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:04.600895 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:07.964417 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:07.964389 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:09.232209 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:09.232168 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:09.840233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:09.840201 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:10.074806 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:10.074771 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:10.412933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:10.412888 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:13.951071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:13.951039 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:14.198572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.198539 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:14.794707 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.794673 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f"] Nov 03 21:54:14.795200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795187 5931 cpu_manager.go:414] "containerMap: removing container" podUID="56cf5107-7bdd-474c-8717-6cbf5dffc680" containerName="maas-api" Nov 03 21:54:14.795246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795202 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56cf5107-7bdd-474c-8717-6cbf5dffc680" containerName="maas-api" Nov 03 21:54:14.795246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795215 5931 cpu_manager.go:414] "containerMap: removing container" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" Nov 03 21:54:14.795246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795220 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" Nov 03 21:54:14.795246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795240 5931 cpu_manager.go:414] "containerMap: removing container" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" Nov 03 21:54:14.795246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795245 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" Nov 03 21:54:14.795399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795324 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="56cf5107-7bdd-474c-8717-6cbf5dffc680" containerName="maas-api" Nov 03 21:54:14.795399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795334 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" Nov 03 21:54:14.795399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.795343 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9655b3a-b651-41f8-91e9-3ab609c61a79" containerName="istio-proxy" Nov 03 21:54:14.798852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.798835 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.890712 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.890682 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.890861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.890716 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68dz\" (UniqueName: \"kubernetes.io/projected/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-kube-api-access-n68dz\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.890861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.890743 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-metrics-certs\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.890978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.890900 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.901856 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.901833 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c993b8-serving-cert\"" Nov 03 21:54:14.904714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.904696 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f"] Nov 03 21:54:14.991811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.991784 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.992250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.991823 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68dz\" (UniqueName: \"kubernetes.io/projected/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-kube-api-access-n68dz\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.992250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.991861 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-metrics-certs\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.992250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.991927 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.992250 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:54:14.992088 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-serving-cert: secret "maas-api-e2e-praxis-a-c993b8-serving-cert" not found Nov 03 21:54:14.992250 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:54:14.992148 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls podName:aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d nodeName:}" failed. No retries permitted until 2026-11-03 21:54:15.492126725 +0000 UTC m=+2323.102974532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls") pod "maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" (UID: "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d") : secret "maas-api-e2e-praxis-a-c993b8-serving-cert" not found Nov 03 21:54:14.992640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.992597 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:14.994465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:14.994443 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-metrics-certs\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:15.051104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:15.051030 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68dz\" (UniqueName: \"kubernetes.io/projected/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-kube-api-access-n68dz\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:15.075180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:15.075155 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:15.339311 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:15.339243 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:15.496659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:15.496600 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:15.499010 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:15.498990 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:15.711693 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:15.711667 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:16.097543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:16.097515 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f"] Nov 03 21:54:16.099001 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:54:16.098976 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebc81d1_e4d2_4526_95d5_b4b10b0d3f7d.slice/crio-2726572d3d847187af966c75f9e6c6b509d2aeceab848c459c8f8145e22d825f WatchSource:0}: Error finding container 2726572d3d847187af966c75f9e6c6b509d2aeceab848c459c8f8145e22d825f: Status 404 returned error can't find the container with id 2726572d3d847187af966c75f9e6c6b509d2aeceab848c459c8f8145e22d825f Nov 03 21:54:16.100502 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:16.100481 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:54:16.471512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:16.471447 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" event={"ID":"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d","Type":"ContainerStarted","Data":"2726572d3d847187af966c75f9e6c6b509d2aeceab848c459c8f8145e22d825f"} Nov 03 21:54:17.466005 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.465976 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:17.493024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.492942 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" event={"ID":"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d","Type":"ContainerStarted","Data":"0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457"} Nov 03 21:54:17.493473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.493445 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:17.608439 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.608386 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" podStartSLOduration=2.798007884 podStartE2EDuration="3.60837236s" totalImagesPullingTime="810.364476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:17.60616284 +0000 UTC m=+2325.217010637" watchObservedRunningTime="2026-11-03 21:54:17.60837236 +0000 UTC m=+2325.219220173" Nov 03 21:54:17.732691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.732593 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:17.781569 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.781543 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv"] Nov 03 21:54:17.785422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.785407 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:17.838861 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.838826 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-292cf1-dockercfg-hzrtc\"" Nov 03 21:54:17.839035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.838951 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-292cf1\"" Nov 03 21:54:17.842179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.842160 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv"] Nov 03 21:54:17.923672 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.923646 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72z9\" (UniqueName: \"kubernetes.io/projected/c819184f-9e18-43c1-a6a8-8633597ab352-kube-api-access-x72z9\") pod \"payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:17.923802 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.923688 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c819184f-9e18-43c1-a6a8-8633597ab352-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:17.929599 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.929574 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s"] Nov 03 21:54:17.933482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.933464 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:17.991714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:17.991600 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s"] Nov 03 21:54:18.024971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.024932 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72z9\" (UniqueName: \"kubernetes.io/projected/c819184f-9e18-43c1-a6a8-8633597ab352-kube-api-access-x72z9\") pod \"payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:18.025150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.024994 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c819184f-9e18-43c1-a6a8-8633597ab352-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:18.025150 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.025104 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8nb\" (UniqueName: \"kubernetes.io/projected/ba3bdfed-684a-49a2-9978-79af19f96c24-kube-api-access-jg8nb\") pod \"payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:18.025279 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.025148 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3bdfed-684a-49a2-9978-79af19f96c24-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:18.025902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.025875 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c819184f-9e18-43c1-a6a8-8633597ab352-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:18.080936 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.080908 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72z9\" (UniqueName: \"kubernetes.io/projected/c819184f-9e18-43c1-a6a8-8633597ab352-kube-api-access-x72z9\") pod \"payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:18.107115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.107088 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:18.126433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.126413 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8nb\" (UniqueName: \"kubernetes.io/projected/ba3bdfed-684a-49a2-9978-79af19f96c24-kube-api-access-jg8nb\") pod \"payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:18.126512 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.126447 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3bdfed-684a-49a2-9978-79af19f96c24-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:18.127315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.127292 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3bdfed-684a-49a2-9978-79af19f96c24-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:18.183078 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.183049 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8nb\" (UniqueName: \"kubernetes.io/projected/ba3bdfed-684a-49a2-9978-79af19f96c24-kube-api-access-jg8nb\") pod \"payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:18.247045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.246979 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:18.479844 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.479819 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv"] Nov 03 21:54:18.482145 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:54:18.482115 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc819184f_9e18_43c1_a6a8_8633597ab352.slice/crio-38afded70373e50bb33da5d4ba8d49f48a416d93c0a7ab9004090e9f8086282f WatchSource:0}: Error finding container 38afded70373e50bb33da5d4ba8d49f48a416d93c0a7ab9004090e9f8086282f: Status 404 returned error can't find the container with id 38afded70373e50bb33da5d4ba8d49f48a416d93c0a7ab9004090e9f8086282f Nov 03 21:54:18.511073 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.510998 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" event={"ID":"c819184f-9e18-43c1-a6a8-8633597ab352","Type":"ContainerStarted","Data":"38afded70373e50bb33da5d4ba8d49f48a416d93c0a7ab9004090e9f8086282f"} Nov 03 21:54:18.625325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:18.625304 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s"] Nov 03 21:54:18.627448 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:54:18.627415 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3bdfed_684a_49a2_9978_79af19f96c24.slice/crio-91225cd1cbf5249956d674fe486a8f5be57687339527509e49ba9e5b1e9ed0e4 WatchSource:0}: Error finding container 91225cd1cbf5249956d674fe486a8f5be57687339527509e49ba9e5b1e9ed0e4: Status 404 returned error can't find the container with id 91225cd1cbf5249956d674fe486a8f5be57687339527509e49ba9e5b1e9ed0e4 Nov 03 21:54:19.509265 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:19.509236 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" event={"ID":"c819184f-9e18-43c1-a6a8-8633597ab352","Type":"ContainerStarted","Data":"f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c"} Nov 03 21:54:19.509662 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:19.509291 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:19.511006 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:19.510983 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" event={"ID":"ba3bdfed-684a-49a2-9978-79af19f96c24","Type":"ContainerStarted","Data":"a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104"} Nov 03 21:54:19.511104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:19.511021 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" event={"ID":"ba3bdfed-684a-49a2-9978-79af19f96c24","Type":"ContainerStarted","Data":"91225cd1cbf5249956d674fe486a8f5be57687339527509e49ba9e5b1e9ed0e4"} Nov 03 21:54:19.511104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:19.511052 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:19.624385 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:19.624338 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" podStartSLOduration=2.624324159 podStartE2EDuration="2.624324159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:19.621725797 +0000 UTC m=+2327.232573614" watchObservedRunningTime="2026-11-03 21:54:19.624324159 +0000 UTC m=+2327.235171974" Nov 03 21:54:19.739128 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:19.739063 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" podStartSLOduration=2.739049574 podStartE2EDuration="2.739049574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:19.733855016 +0000 UTC m=+2327.344702832" watchObservedRunningTime="2026-11-03 21:54:19.739049574 +0000 UTC m=+2327.349897507" Nov 03 21:54:20.099494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:20.099465 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:20.313145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:20.313113 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:20.546492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:20.546420 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:23.505800 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:23.505774 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:54:25.664989 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:25.664958 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:30.519270 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:30.519245 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:54:30.519667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:30.519377 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:54:30.823449 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:30.823379 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:33.068188 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:54:33.068151 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:54:33.289038 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:33.289009 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:33.511442 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:33.511420 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:36.350114 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.350080 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv"] Nov 03 21:54:36.350542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.350284 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" podUID="191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" containerName="maas-api" containerID="cri-o://f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02" gracePeriod=30 Nov 03 21:54:36.561550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.561515 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:36.606824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.606771 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:54:36.616780 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.616752 5931 generic.go:356] "Generic (PLEG): container finished" podID="191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" containerID="f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02" exitCode=0 Nov 03 21:54:36.616913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.616802 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" Nov 03 21:54:36.616913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.616827 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" event={"ID":"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a","Type":"ContainerDied","Data":"f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02"} Nov 03 21:54:36.616913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.616879 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv" event={"ID":"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a","Type":"ContainerDied","Data":"fa63d30a0ac79ee8d4eec3344d9ebaf2d155d2a596d590adffe98aab2caf0b28"} Nov 03 21:54:36.616913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.616899 5931 scope.go:122] "RemoveContainer" containerID="f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02" Nov 03 21:54:36.628912 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.628888 5931 scope.go:122] "RemoveContainer" containerID="f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02" Nov 03 21:54:36.629347 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:54:36.629324 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02\": container with ID starting with f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02 not found: ID does not exist" containerID="f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02" Nov 03 21:54:36.629410 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.629359 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02"} err="failed to get container status \"f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02\": rpc error: code = NotFound desc = could not find container \"f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02\": container with ID starting with f225a4b9d67aac2a5c8fca2418b127741f36c95686ad419b1842e2e4c8519b02 not found: ID does not exist" Nov 03 21:54:36.700364 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.700345 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-openshift-service-ca\") pod \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " Nov 03 21:54:36.700453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.700393 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls\") pod \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " Nov 03 21:54:36.700537 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.700522 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-kube-api-access-jz5z5\" (UniqueName: \"kubernetes.io/projected/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-kube-api-access-jz5z5\") pod \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " Nov 03 21:54:36.700598 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.700583 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-metrics-certs\") pod \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\" (UID: \"191b4e57-e4be-4a3b-aa25-9879fd8dbd4a\") " Nov 03 21:54:36.700751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.700732 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-openshift-service-ca" pod "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" (UID: "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:36.701022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.701004 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:54:36.702643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.702586 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-metrics-certs" pod "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" (UID: "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:36.702643 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.702636 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls" pod "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" (UID: "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:36.702759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.702688 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-kube-api-access-jz5z5" pod "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" (UID: "191b4e57-e4be-4a3b-aa25-9879fd8dbd4a"). InnerVolumeSpecName "kube-api-access-jz5z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:36.801918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.801888 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz5z5\" (UniqueName: \"kubernetes.io/projected/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-kube-api-access-jz5z5\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:54:36.802066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.801922 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:54:36.802066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:36.801937 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:54:37.039359 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:37.039329 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv"] Nov 03 21:54:37.043383 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:37.043358 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ae2f6f-5ff6d8dff6-552jv"] Nov 03 21:54:38.506474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:38.506443 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" path="/var/lib/kubelet/pods/191b4e57-e4be-4a3b-aa25-9879fd8dbd4a/volumes" Nov 03 21:54:38.725890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:38.725855 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:39.125539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:39.125507 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:43.504218 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.504186 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv"] Nov 03 21:54:43.504774 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.504760 5931 cpu_manager.go:414] "containerMap: removing container" podUID="191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" containerName="maas-api" Nov 03 21:54:43.504829 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.504776 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" containerName="maas-api" Nov 03 21:54:43.504903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.504894 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="191b4e57-e4be-4a3b-aa25-9879fd8dbd4a" containerName="maas-api" Nov 03 21:54:43.514138 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.514113 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.564323 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.564283 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.564468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.564444 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03322591-3d96-4ca7-abda-0497f9c6381d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.564531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.564512 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.564589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.564551 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpddt\" (UniqueName: \"kubernetes.io/projected/03322591-3d96-4ca7-abda-0497f9c6381d-kube-api-access-lpddt\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.564915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.564896 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv"] Nov 03 21:54:43.610815 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.610792 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6ddbba-serving-cert\"" Nov 03 21:54:43.665240 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.665219 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.665346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.665281 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03322591-3d96-4ca7-abda-0497f9c6381d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.665346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.665328 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.665455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.665370 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpddt\" (UniqueName: \"kubernetes.io/projected/03322591-3d96-4ca7-abda-0497f9c6381d-kube-api-access-lpddt\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.666312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.666285 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03322591-3d96-4ca7-abda-0497f9c6381d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.668224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.668204 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.670015 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.669989 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.721247 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.721220 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpddt\" (UniqueName: \"kubernetes.io/projected/03322591-3d96-4ca7-abda-0497f9c6381d-kube-api-access-lpddt\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:43.828482 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:43.828429 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:44.015752 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:44.015700 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:44.200718 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:44.200691 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv"] Nov 03 21:54:44.202924 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:54:44.202899 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03322591_3d96_4ca7_abda_0497f9c6381d.slice/crio-6a4f474a87dc410f632592ccb1c0b6f78ec54ea9aba97fdb6f814e77c0bbd8b4 WatchSource:0}: Error finding container 6a4f474a87dc410f632592ccb1c0b6f78ec54ea9aba97fdb6f814e77c0bbd8b4: Status 404 returned error can't find the container with id 6a4f474a87dc410f632592ccb1c0b6f78ec54ea9aba97fdb6f814e77c0bbd8b4 Nov 03 21:54:44.665854 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:44.665822 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" event={"ID":"03322591-3d96-4ca7-abda-0497f9c6381d","Type":"ContainerStarted","Data":"6a4f474a87dc410f632592ccb1c0b6f78ec54ea9aba97fdb6f814e77c0bbd8b4"} Nov 03 21:54:45.022517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:45.022489 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:45.284934 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:45.284865 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:45.534885 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:45.534848 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:45.673224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:45.673185 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" event={"ID":"03322591-3d96-4ca7-abda-0497f9c6381d","Type":"ContainerStarted","Data":"29cbb5546e901a749ed20628c1eb10cc8201e63bdc0d729040a2949be7273435"} Nov 03 21:54:45.673689 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:45.673339 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:45.796154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:45.796117 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" podStartSLOduration=2.008964369 podStartE2EDuration="2.796104472s" totalImagesPullingTime="787.140103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:45.794470504 +0000 UTC m=+2353.405318320" watchObservedRunningTime="2026-11-03 21:54:45.796104472 +0000 UTC m=+2353.406952286" Nov 03 21:54:47.340381 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:47.340346 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:47.568941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:47.568907 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:48.161909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:48.161883 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:48.815190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:48.815156 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:49.180123 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:49.180094 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:49.408584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:49.408549 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:51.685945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:51.685913 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:54:54.666669 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:54.666602 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:54.891255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:54.891179 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:55.258275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:55.258203 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:59.610062 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:59.610011 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:59.835372 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:54:59.835285 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:00.571554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:00.571522 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:03.068228 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:03.068192 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:55:05.665283 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:05.665239 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:10.507850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:10.507815 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:15.073853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:15.073819 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:15.301389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:15.301354 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:19.932661 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:19.932630 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:24.740405 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:24.740367 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:25.300605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:25.300574 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:29.031473 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.031443 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p"] Nov 03 21:55:29.031949 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.031632 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" podUID="da556241-a4a1-4b71-bac1-c3671f91a415" containerName="maas-api" containerID="cri-o://3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4" gracePeriod=30 Nov 03 21:55:29.214411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.213149 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:29.292423 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.292359 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:55:29.416096 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.416060 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-metrics-certs\") pod \"da556241-a4a1-4b71-bac1-c3671f91a415\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " Nov 03 21:55:29.416275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.416139 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da556241-a4a1-4b71-bac1-c3671f91a415-kube-api-access-n98xg\" (UniqueName: \"kubernetes.io/projected/da556241-a4a1-4b71-bac1-c3671f91a415-kube-api-access-n98xg\") pod \"da556241-a4a1-4b71-bac1-c3671f91a415\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " Nov 03 21:55:29.416275 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.416207 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da556241-a4a1-4b71-bac1-c3671f91a415-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da556241-a4a1-4b71-bac1-c3671f91a415-openshift-service-ca\") pod \"da556241-a4a1-4b71-bac1-c3671f91a415\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " Nov 03 21:55:29.416391 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.416283 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\") pod \"da556241-a4a1-4b71-bac1-c3671f91a415\" (UID: \"da556241-a4a1-4b71-bac1-c3671f91a415\") " Nov 03 21:55:29.417469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.417427 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da556241-a4a1-4b71-bac1-c3671f91a415-openshift-service-ca" pod "da556241-a4a1-4b71-bac1-c3671f91a415" (UID: "da556241-a4a1-4b71-bac1-c3671f91a415"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:29.419388 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.419363 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da556241-a4a1-4b71-bac1-c3671f91a415-kube-api-access-n98xg" pod "da556241-a4a1-4b71-bac1-c3671f91a415" (UID: "da556241-a4a1-4b71-bac1-c3671f91a415"). InnerVolumeSpecName "kube-api-access-n98xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:29.420021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.419987 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls" pod "da556241-a4a1-4b71-bac1-c3671f91a415" (UID: "da556241-a4a1-4b71-bac1-c3671f91a415"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:29.420762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.420739 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-metrics-certs" pod "da556241-a4a1-4b71-bac1-c3671f91a415" (UID: "da556241-a4a1-4b71-bac1-c3671f91a415"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:29.517921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.517893 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n98xg\" (UniqueName: \"kubernetes.io/projected/da556241-a4a1-4b71-bac1-c3671f91a415-kube-api-access-n98xg\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:29.517921 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.517918 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da556241-a4a1-4b71-bac1-c3671f91a415-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:29.518066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.517928 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:29.518066 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.517938 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da556241-a4a1-4b71-bac1-c3671f91a415-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:29.543812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.543752 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj"] Nov 03 21:55:29.543927 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.543908 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" podUID="24b5a385-3dc3-449d-b641-7f4e3b29ecc2" containerName="payload-pre-processing" containerID="cri-o://9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da" gracePeriod=30 Nov 03 21:55:29.727470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.727428 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:29.789354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.789330 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:55:29.922013 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.921987 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-plugins-config-volume\") pod \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " Nov 03 21:55:29.922172 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.922062 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-kube-api-access-wgwxw\" (UniqueName: \"kubernetes.io/projected/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-kube-api-access-wgwxw\") pod \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\" (UID: \"24b5a385-3dc3-449d-b641-7f4e3b29ecc2\") " Nov 03 21:55:29.922329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.922310 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-plugins-config-volume" pod "24b5a385-3dc3-449d-b641-7f4e3b29ecc2" (UID: "24b5a385-3dc3-449d-b641-7f4e3b29ecc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:29.922562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.922535 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:29.924203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.924181 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-kube-api-access-wgwxw" pod "24b5a385-3dc3-449d-b641-7f4e3b29ecc2" (UID: "24b5a385-3dc3-449d-b641-7f4e3b29ecc2"). InnerVolumeSpecName "kube-api-access-wgwxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:29.937221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.937196 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:29.949707 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.949681 5931 generic.go:356] "Generic (PLEG): container finished" podID="24b5a385-3dc3-449d-b641-7f4e3b29ecc2" containerID="9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da" exitCode=0 Nov 03 21:55:29.949812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.949733 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" Nov 03 21:55:29.949812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.949763 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" event={"ID":"24b5a385-3dc3-449d-b641-7f4e3b29ecc2","Type":"ContainerDied","Data":"9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da"} Nov 03 21:55:29.949915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.949814 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj" event={"ID":"24b5a385-3dc3-449d-b641-7f4e3b29ecc2","Type":"ContainerDied","Data":"d05b6af80d24ddbdba94c0a636d92747ee211e2afa899dc6a8ba2b351df7b0d4"} Nov 03 21:55:29.949915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.949846 5931 scope.go:122] "RemoveContainer" containerID="9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da" Nov 03 21:55:29.951346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.951319 5931 generic.go:356] "Generic (PLEG): container finished" podID="da556241-a4a1-4b71-bac1-c3671f91a415" containerID="3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4" exitCode=0 Nov 03 21:55:29.951438 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.951367 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" Nov 03 21:55:29.951438 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.951373 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" event={"ID":"da556241-a4a1-4b71-bac1-c3671f91a415","Type":"ContainerDied","Data":"3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4"} Nov 03 21:55:29.951438 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.951411 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p" event={"ID":"da556241-a4a1-4b71-bac1-c3671f91a415","Type":"ContainerDied","Data":"95c7709acc1da44b7d8898795a2b354c907bb658ebc86378976cb64227293531"} Nov 03 21:55:29.960245 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.960185 5931 scope.go:122] "RemoveContainer" containerID="9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da" Nov 03 21:55:29.960515 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:29.960492 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da\": container with ID starting with 9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da not found: ID does not exist" containerID="9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da" Nov 03 21:55:29.960589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.960524 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da"} err="failed to get container status \"9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da\": rpc error: code = NotFound desc = could not find container \"9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da\": container with ID starting with 9adcf4824e897725ab83007db5f449a25a798a882b666d3992d1ebccbccd78da not found: ID does not exist" Nov 03 21:55:29.960589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.960540 5931 scope.go:122] "RemoveContainer" containerID="3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4" Nov 03 21:55:29.969451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.969430 5931 scope.go:122] "RemoveContainer" containerID="3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4" Nov 03 21:55:29.969738 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:29.969721 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4\": container with ID starting with 3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4 not found: ID does not exist" containerID="3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4" Nov 03 21:55:29.969793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:29.969745 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4"} err="failed to get container status \"3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4\": rpc error: code = NotFound desc = could not find container \"3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4\": container with ID starting with 3efe65dc11dd2088efa9702eeae9d5525a455b67e9bba760a47c860f06fdf4e4 not found: ID does not exist" Nov 03 21:55:30.023244 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.023214 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgwxw\" (UniqueName: \"kubernetes.io/projected/24b5a385-3dc3-449d-b641-7f4e3b29ecc2-kube-api-access-wgwxw\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:30.120531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.120493 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p"] Nov 03 21:55:30.127528 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.127490 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c3bed-5c8898874f-jnh5p"] Nov 03 21:55:30.235889 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.235829 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj"] Nov 03 21:55:30.241221 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.241197 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c3bed-5499b8449f-kx9dj"] Nov 03 21:55:30.506193 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.506132 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b5a385-3dc3-449d-b641-7f4e3b29ecc2" path="/var/lib/kubelet/pods/24b5a385-3dc3-449d-b641-7f4e3b29ecc2/volumes" Nov 03 21:55:30.506539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.506524 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da556241-a4a1-4b71-bac1-c3671f91a415" path="/var/lib/kubelet/pods/da556241-a4a1-4b71-bac1-c3671f91a415/volumes" Nov 03 21:55:30.651684 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.651647 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:30.903001 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:30.902968 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:31.113741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:31.113708 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:31.322326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:31.322247 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:31.857308 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:31.857276 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:32.083589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.083555 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:32.484749 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.484711 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx"] Nov 03 21:55:32.485422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.485401 5931 cpu_manager.go:414] "containerMap: removing container" podUID="da556241-a4a1-4b71-bac1-c3671f91a415" containerName="maas-api" Nov 03 21:55:32.485526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.485423 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da556241-a4a1-4b71-bac1-c3671f91a415" containerName="maas-api" Nov 03 21:55:32.485526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.485459 5931 cpu_manager.go:414] "containerMap: removing container" podUID="24b5a385-3dc3-449d-b641-7f4e3b29ecc2" containerName="payload-pre-processing" Nov 03 21:55:32.485526 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.485467 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b5a385-3dc3-449d-b641-7f4e3b29ecc2" containerName="payload-pre-processing" Nov 03 21:55:32.485740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.485654 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="da556241-a4a1-4b71-bac1-c3671f91a415" containerName="maas-api" Nov 03 21:55:32.485740 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.485673 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b5a385-3dc3-449d-b641-7f4e3b29ecc2" containerName="payload-pre-processing" Nov 03 21:55:32.490646 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.490625 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.542309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.542287 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cd2a9f-openshift-default-dockercfg-46ngq\"" Nov 03 21:55:32.595108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.595080 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx"] Nov 03 21:55:32.641056 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.641022 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:32.649756 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.649734 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-podinfo\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.649911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.649760 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-token\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.649911 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.649797 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-data\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.650031 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.649995 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-credential-socket\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.650079 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.650047 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-certs\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.650121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.650105 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istiod-ca-cert\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.650176 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.650164 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-socket\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.650228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.650183 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6jr\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-kube-api-access-4c6jr\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.650228 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.650211 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-envoy\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751016 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-credential-socket\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751050 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-certs\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751067 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istiod-ca-cert\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751096 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-socket\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751113 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6jr\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-kube-api-access-4c6jr\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751146 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-envoy\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751242 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-podinfo\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751295 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751262 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-token\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751296 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-data\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751438 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-certs\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751553 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751507 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-credential-socket\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751716 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751551 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-socket\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751716 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751630 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-data\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.751917 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.751892 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istiod-ca-cert\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.753701 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.753677 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-envoy\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.753864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.753847 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-podinfo\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.811538 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.811513 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-token\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:32.811780 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:32.811761 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6jr\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-kube-api-access-4c6jr\") pod \"e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:33.068575 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:33.068493 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:55:33.080868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.080841 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:33.103922 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.103903 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:33.532570 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.532547 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx"] Nov 03 21:55:33.535158 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:55:33.535128 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9783f1_ed5d_440c_99c7_1c9b0f8a7791.slice/crio-02513e4daff6d1002212d4eac465f86343fba06dc1c2a95b8e0293ed03ca71bd WatchSource:0}: Error finding container 02513e4daff6d1002212d4eac465f86343fba06dc1c2a95b8e0293ed03ca71bd: Status 404 returned error can't find the container with id 02513e4daff6d1002212d4eac465f86343fba06dc1c2a95b8e0293ed03ca71bd Nov 03 21:55:33.537494 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.537459 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:55:33.537606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.537539 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:55:33.537606 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.537583 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:55:33.979554 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.979521 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" event={"ID":"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791","Type":"ContainerStarted","Data":"6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab"} Nov 03 21:55:33.979754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:33.979584 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" event={"ID":"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791","Type":"ContainerStarted","Data":"02513e4daff6d1002212d4eac465f86343fba06dc1c2a95b8e0293ed03ca71bd"} Nov 03 21:55:34.099188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:34.099146 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" podStartSLOduration=2.099133522 podStartE2EDuration="2.099133522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:34.097794246 +0000 UTC m=+2401.708642062" watchObservedRunningTime="2026-11-03 21:55:34.099133522 +0000 UTC m=+2401.709981336" Nov 03 21:55:34.104864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:34.104847 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:34.109933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:34.109912 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:34.986574 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:34.986539 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:34.987293 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:34.987275 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:55:35.611384 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:35.611360 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:55:35.621914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:35.621893 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 21:55:36.417241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:36.417207 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:36.722758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:36.722690 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs"] Nov 03 21:55:36.722923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:36.722900 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" containerID="cri-o://b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b" gracePeriod=30 Nov 03 21:55:36.768736 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:36.768710 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47"] Nov 03 21:55:36.900250 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:36.900210 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:37.131958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:37.131924 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:37.357263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:37.357232 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:40.375016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:40.374983 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:40.590658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:40.590603 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:41.985845 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:41.985795 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:55:42.030445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.030416 5931 generic.go:356] "Generic (PLEG): container finished" podID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerID="b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b" exitCode=0 Nov 03 21:55:42.030590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.030466 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" Nov 03 21:55:42.030590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.030501 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" event={"ID":"b528a122-28ef-4c30-a656-a98ed38cddb6","Type":"ContainerDied","Data":"b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b"} Nov 03 21:55:42.030590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.030547 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs" event={"ID":"b528a122-28ef-4c30-a656-a98ed38cddb6","Type":"ContainerDied","Data":"5b5beb1452efec6b48f22330608f6dd002ab0caefd47e9c9fd84f504ab0b319d"} Nov 03 21:55:42.030590 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.030572 5931 scope.go:122] "RemoveContainer" containerID="b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b" Nov 03 21:55:42.040335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.040312 5931 scope.go:122] "RemoveContainer" containerID="b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b" Nov 03 21:55:42.040597 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:42.040580 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b\": container with ID starting with b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b not found: ID does not exist" containerID="b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b" Nov 03 21:55:42.040681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.040626 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b"} err="failed to get container status \"b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b\": rpc error: code = NotFound desc = could not find container \"b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b\": container with ID starting with b80fa3e8389785af37b80c8ae2d3ba5bdd3edbc40b56d5bf83dfc481f025286b not found: ID does not exist" Nov 03 21:55:42.146743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.146715 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b528a122-28ef-4c30-a656-a98ed38cddb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b528a122-28ef-4c30-a656-a98ed38cddb6-istiod-ca-cert\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.146901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.146776 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-certs\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.146901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.146831 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-socket\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.146901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.146879 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-envoy\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.147086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.146903 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-token\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-token\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.147086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.146957 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-kube-api-access-zvrjl\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-kube-api-access-zvrjl\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.147086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.146991 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-data\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.147086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147068 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-credential-socket\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.147307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147108 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b528a122-28ef-4c30-a656-a98ed38cddb6-istiod-ca-cert" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:42.147307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147139 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-certs" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:42.147307 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147181 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-podinfo\") pod \"b528a122-28ef-4c30-a656-a98ed38cddb6\" (UID: \"b528a122-28ef-4c30-a656-a98ed38cddb6\") " Nov 03 21:55:42.147465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147344 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-data" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:42.147524 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147504 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-socket" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:42.147720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147665 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.147863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147769 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b528a122-28ef-4c30-a656-a98ed38cddb6-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.147918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147864 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.147918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147796 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-credential-socket" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:42.147918 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.147878 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.149364 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.149342 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-token" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:42.149542 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.149524 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-kube-api-access-zvrjl" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "kube-api-access-zvrjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:42.150053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.150033 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-podinfo" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:55:42.150144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.150030 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-envoy" pod "b528a122-28ef-4c30-a656-a98ed38cddb6" (UID: "b528a122-28ef-4c30-a656-a98ed38cddb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:42.248919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.248853 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.248919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.248877 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.248919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.248887 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.248919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.248895 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.248919 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.248905 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrjl\" (UniqueName: \"kubernetes.io/projected/b528a122-28ef-4c30-a656-a98ed38cddb6-kube-api-access-zvrjl\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:42.451165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.451140 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs"] Nov 03 21:55:42.455334 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.455312 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-pbscs"] Nov 03 21:55:42.507953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:42.507897 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" path="/var/lib/kubelet/pods/b528a122-28ef-4c30-a656-a98ed38cddb6/volumes" Nov 03 21:55:43.417336 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:43.417307 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:48.284457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:48.284419 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:49.505728 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:49.505701 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:51.790759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:51.790720 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" podUID="aea49457-8cd1-4b84-a6f2-708ba489ad06" containerName="main" containerID="cri-o://40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e" gracePeriod=45 Nov 03 21:55:51.837214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:51.837186 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" podUID="aea49457-8cd1-4b84-a6f2-708ba489ad06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.135:8000/ready\": dial tcp 10.134.0.135:8000: connect: connection refused" Nov 03 21:55:52.057058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.057005 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:55:52.097022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.096991 5931 generic.go:356] "Generic (PLEG): container finished" podID="aea49457-8cd1-4b84-a6f2-708ba489ad06" containerID="40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e" exitCode=0 Nov 03 21:55:52.097124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.097032 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" Nov 03 21:55:52.097124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.097074 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" event={"ID":"aea49457-8cd1-4b84-a6f2-708ba489ad06","Type":"ContainerDied","Data":"40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e"} Nov 03 21:55:52.097124 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.097118 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47" event={"ID":"aea49457-8cd1-4b84-a6f2-708ba489ad06","Type":"ContainerDied","Data":"f21ed6fc16624a7ce2d6d381a596a903025f9d938016a29f53539e93857936a3"} Nov 03 21:55:52.097220 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.097142 5931 scope.go:122] "RemoveContainer" containerID="40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e" Nov 03 21:55:52.106171 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.106155 5931 scope.go:122] "RemoveContainer" containerID="40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e" Nov 03 21:55:52.106435 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:52.106418 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e\": container with ID starting with 40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e not found: ID does not exist" containerID="40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e" Nov 03 21:55:52.106488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.106444 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e"} err="failed to get container status \"40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e\": rpc error: code = NotFound desc = could not find container \"40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e\": container with ID starting with 40b54cb04e2abcaeaf02a0a41105d794cca990366179fd58327fb15e8ca8a65e not found: ID does not exist" Nov 03 21:55:52.125863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.125845 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea49457-8cd1-4b84-a6f2-708ba489ad06-kube-api-access-jm69j\" (UniqueName: \"kubernetes.io/projected/aea49457-8cd1-4b84-a6f2-708ba489ad06-kube-api-access-jm69j\") pod \"aea49457-8cd1-4b84-a6f2-708ba489ad06\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " Nov 03 21:55:52.125953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.125874 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-tmp-dir\") pod \"aea49457-8cd1-4b84-a6f2-708ba489ad06\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " Nov 03 21:55:52.125953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.125923 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-model-cache\") pod \"aea49457-8cd1-4b84-a6f2-708ba489ad06\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " Nov 03 21:55:52.126071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.125953 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aea49457-8cd1-4b84-a6f2-708ba489ad06-tls-certs\" (UniqueName: \"kubernetes.io/secret/aea49457-8cd1-4b84-a6f2-708ba489ad06-tls-certs\") pod \"aea49457-8cd1-4b84-a6f2-708ba489ad06\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " Nov 03 21:55:52.126071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.125991 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-home\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-home\") pod \"aea49457-8cd1-4b84-a6f2-708ba489ad06\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " Nov 03 21:55:52.126071 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.126020 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-dshm\") pod \"aea49457-8cd1-4b84-a6f2-708ba489ad06\" (UID: \"aea49457-8cd1-4b84-a6f2-708ba489ad06\") " Nov 03 21:55:52.126235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.126211 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-tmp-dir" pod "aea49457-8cd1-4b84-a6f2-708ba489ad06" (UID: "aea49457-8cd1-4b84-a6f2-708ba489ad06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:52.126287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.126225 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-model-cache" pod "aea49457-8cd1-4b84-a6f2-708ba489ad06" (UID: "aea49457-8cd1-4b84-a6f2-708ba489ad06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:52.126287 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.126234 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-home" pod "aea49457-8cd1-4b84-a6f2-708ba489ad06" (UID: "aea49457-8cd1-4b84-a6f2-708ba489ad06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:52.126435 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.126420 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:52.126492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.126437 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:52.126492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.126447 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:52.128117 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.128095 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea49457-8cd1-4b84-a6f2-708ba489ad06-tls-certs" pod "aea49457-8cd1-4b84-a6f2-708ba489ad06" (UID: "aea49457-8cd1-4b84-a6f2-708ba489ad06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:52.128190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.128125 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea49457-8cd1-4b84-a6f2-708ba489ad06-kube-api-access-jm69j" pod "aea49457-8cd1-4b84-a6f2-708ba489ad06" (UID: "aea49457-8cd1-4b84-a6f2-708ba489ad06"). InnerVolumeSpecName "kube-api-access-jm69j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:52.128190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.128136 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-dshm" pod "aea49457-8cd1-4b84-a6f2-708ba489ad06" (UID: "aea49457-8cd1-4b84-a6f2-708ba489ad06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:55:52.227501 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.227464 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aea49457-8cd1-4b84-a6f2-708ba489ad06-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:52.227658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.227503 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm69j\" (UniqueName: \"kubernetes.io/projected/aea49457-8cd1-4b84-a6f2-708ba489ad06-kube-api-access-jm69j\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:52.227658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.227521 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aea49457-8cd1-4b84-a6f2-708ba489ad06-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:52.518694 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.518661 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47"] Nov 03 21:55:52.521234 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:52.521213 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1c3bed/ipp-route-1c3bed-kserve-777c5f6d9c-cvs47"] Nov 03 21:55:53.017223 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:53.017192 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:54.505413 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:54.505380 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea49457-8cd1-4b84-a6f2-708ba489ad06" path="/var/lib/kubelet/pods/aea49457-8cd1-4b84-a6f2-708ba489ad06/volumes" Nov 03 21:55:55.984584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:55.984547 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv"] Nov 03 21:55:55.985149 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:55.984810 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" podUID="03322591-3d96-4ca7-abda-0497f9c6381d" containerName="maas-api" containerID="cri-o://29cbb5546e901a749ed20628c1eb10cc8201e63bdc0d729040a2949be7273435" gracePeriod=30 Nov 03 21:55:56.124184 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.124147 5931 generic.go:356] "Generic (PLEG): container finished" podID="03322591-3d96-4ca7-abda-0497f9c6381d" containerID="29cbb5546e901a749ed20628c1eb10cc8201e63bdc0d729040a2949be7273435" exitCode=0 Nov 03 21:55:56.124318 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.124214 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" event={"ID":"03322591-3d96-4ca7-abda-0497f9c6381d","Type":"ContainerDied","Data":"29cbb5546e901a749ed20628c1eb10cc8201e63bdc0d729040a2949be7273435"} Nov 03 21:55:56.242364 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.242307 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:55:56.264683 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.264659 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-metrics-certs\") pod \"03322591-3d96-4ca7-abda-0497f9c6381d\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " Nov 03 21:55:56.264784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.264773 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03322591-3d96-4ca7-abda-0497f9c6381d-kube-api-access-lpddt\" (UniqueName: \"kubernetes.io/projected/03322591-3d96-4ca7-abda-0497f9c6381d-kube-api-access-lpddt\") pod \"03322591-3d96-4ca7-abda-0497f9c6381d\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " Nov 03 21:55:56.264827 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.264801 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-maas-api-tls\") pod \"03322591-3d96-4ca7-abda-0497f9c6381d\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " Nov 03 21:55:56.264868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.264844 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03322591-3d96-4ca7-abda-0497f9c6381d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03322591-3d96-4ca7-abda-0497f9c6381d-openshift-service-ca\") pod \"03322591-3d96-4ca7-abda-0497f9c6381d\" (UID: \"03322591-3d96-4ca7-abda-0497f9c6381d\") " Nov 03 21:55:56.265286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.265239 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03322591-3d96-4ca7-abda-0497f9c6381d-openshift-service-ca" pod "03322591-3d96-4ca7-abda-0497f9c6381d" (UID: "03322591-3d96-4ca7-abda-0497f9c6381d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:56.267017 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.266989 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-maas-api-tls" pod "03322591-3d96-4ca7-abda-0497f9c6381d" (UID: "03322591-3d96-4ca7-abda-0497f9c6381d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:56.267387 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.267359 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-metrics-certs" pod "03322591-3d96-4ca7-abda-0497f9c6381d" (UID: "03322591-3d96-4ca7-abda-0497f9c6381d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:56.267474 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.267408 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03322591-3d96-4ca7-abda-0497f9c6381d-kube-api-access-lpddt" pod "03322591-3d96-4ca7-abda-0497f9c6381d" (UID: "03322591-3d96-4ca7-abda-0497f9c6381d"). InnerVolumeSpecName "kube-api-access-lpddt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:56.365958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.365927 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpddt\" (UniqueName: \"kubernetes.io/projected/03322591-3d96-4ca7-abda-0497f9c6381d-kube-api-access-lpddt\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:56.365958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.365953 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:56.366127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.365968 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03322591-3d96-4ca7-abda-0497f9c6381d-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:56.366127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:56.365977 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03322591-3d96-4ca7-abda-0497f9c6381d-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:55:57.131327 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:57.131304 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" Nov 03 21:55:57.131769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:57.131302 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv" event={"ID":"03322591-3d96-4ca7-abda-0497f9c6381d","Type":"ContainerDied","Data":"6a4f474a87dc410f632592ccb1c0b6f78ec54ea9aba97fdb6f814e77c0bbd8b4"} Nov 03 21:55:57.131769 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:57.131429 5931 scope.go:122] "RemoveContainer" containerID="29cbb5546e901a749ed20628c1eb10cc8201e63bdc0d729040a2949be7273435" Nov 03 21:55:57.293782 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:57.293758 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv"] Nov 03 21:55:57.342915 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:57.342892 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-fjxrv"] Nov 03 21:55:58.093033 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093007 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt"] Nov 03 21:55:58.093547 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093512 5931 cpu_manager.go:414] "containerMap: removing container" podUID="03322591-3d96-4ca7-abda-0497f9c6381d" containerName="maas-api" Nov 03 21:55:58.093547 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093526 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03322591-3d96-4ca7-abda-0497f9c6381d" containerName="maas-api" Nov 03 21:55:58.093681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093563 5931 cpu_manager.go:414] "containerMap: removing container" podUID="aea49457-8cd1-4b84-a6f2-708ba489ad06" containerName="main" Nov 03 21:55:58.093681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093569 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea49457-8cd1-4b84-a6f2-708ba489ad06" containerName="main" Nov 03 21:55:58.093681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093581 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" Nov 03 21:55:58.093681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093586 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" Nov 03 21:55:58.093824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093683 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="03322591-3d96-4ca7-abda-0497f9c6381d" containerName="maas-api" Nov 03 21:55:58.093824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093695 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea49457-8cd1-4b84-a6f2-708ba489ad06" containerName="main" Nov 03 21:55:58.093824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.093703 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b528a122-28ef-4c30-a656-a98ed38cddb6" containerName="istio-proxy" Nov 03 21:55:58.098326 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.098312 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.181341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.181303 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.181760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.181415 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.181760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.181452 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.181760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.181482 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w57t\" (UniqueName: \"kubernetes.io/projected/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-kube-api-access-2w57t\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.201465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.201443 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-eb3520-serving-cert\"" Nov 03 21:55:58.204057 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.204038 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt"] Nov 03 21:55:58.282737 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.282705 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.282867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.282768 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.282867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.282798 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.282867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.282827 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w57t\" (UniqueName: \"kubernetes.io/projected/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-kube-api-access-2w57t\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.282867 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:58.282841 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-serving-cert: secret "maas-api-e2e-praxis-a-eb3520-serving-cert" not found Nov 03 21:55:58.283070 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:55:58.282906 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls podName:e717b7f8-9269-4af4-be82-ba7a38ee5ca8 nodeName:}" failed. No retries permitted until 2026-11-03 21:55:58.782887941 +0000 UTC m=+2426.393735750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls") pod "maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" (UID: "e717b7f8-9269-4af4-be82-ba7a38ee5ca8") : secret "maas-api-e2e-praxis-a-eb3520-serving-cert" not found Nov 03 21:55:58.283500 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.283479 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.285261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.285239 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.339210 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.339189 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w57t\" (UniqueName: \"kubernetes.io/projected/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-kube-api-access-2w57t\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.377361 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.377302 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:58.506297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.506273 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03322591-3d96-4ca7-abda-0497f9c6381d" path="/var/lib/kubelet/pods/03322591-3d96-4ca7-abda-0497f9c6381d/volumes" Nov 03 21:55:58.605188 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.605165 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:58.789593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.789519 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.791965 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.791946 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:58.838203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:58.838181 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:59.018697 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:59.018675 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:55:59.395493 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:55:59.395470 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt"] Nov 03 21:55:59.397297 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:55:59.397273 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode717b7f8_9269_4af4_be82_ba7a38ee5ca8.slice/crio-db2ff3032aa7727f8c8e211af497c1b4c536e726656c58697d7809bd2bc29ff8 WatchSource:0}: Error finding container db2ff3032aa7727f8c8e211af497c1b4c536e726656c58697d7809bd2bc29ff8: Status 404 returned error can't find the container with id db2ff3032aa7727f8c8e211af497c1b4c536e726656c58697d7809bd2bc29ff8 Nov 03 21:56:00.151663 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:00.151633 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" event={"ID":"e717b7f8-9269-4af4-be82-ba7a38ee5ca8","Type":"ContainerStarted","Data":"db2ff3032aa7727f8c8e211af497c1b4c536e726656c58697d7809bd2bc29ff8"} Nov 03 21:56:01.128268 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:01.128235 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm"] Nov 03 21:56:01.128773 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:01.128422 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" containerID="cri-o://ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0" gracePeriod=30 Nov 03 21:56:01.160658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:01.160630 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" event={"ID":"e717b7f8-9269-4af4-be82-ba7a38ee5ca8","Type":"ContainerStarted","Data":"fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8"} Nov 03 21:56:01.160809 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:01.160680 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:56:01.359651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:01.359565 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" podStartSLOduration=2.560227973 podStartE2EDuration="3.359547899s" totalImagesPullingTime="799.319926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:01.358737607 +0000 UTC m=+2428.969585421" watchObservedRunningTime="2026-11-03 21:56:01.359547899 +0000 UTC m=+2428.970395715" Nov 03 21:56:03.067462 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:03.067427 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:56:03.234739 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:03.234709 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:56:03.234912 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:03.234774 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:56:03.920157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:03.920121 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:04.421115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:04.421079 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:06.382373 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.382351 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:56:06.461068 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461021 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/972de09d-80a9-4a4c-b226-53933aae736f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972de09d-80a9-4a4c-b226-53933aae736f-istio-podinfo\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461081 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-socket\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461103 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-certs\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461126 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-kube-api-access-n5bl7\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-kube-api-access-n5bl7\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461163 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-istio-token\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-istio-token\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461357 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461198 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-credential-socket\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461357 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461265 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-data\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461363 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972de09d-80a9-4a4c-b226-53933aae736f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972de09d-80a9-4a4c-b226-53933aae736f-istiod-ca-cert\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461394 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-envoy\") pod \"972de09d-80a9-4a4c-b226-53933aae736f\" (UID: \"972de09d-80a9-4a4c-b226-53933aae736f\") " Nov 03 21:56:06.461468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461379 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-certs" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:06.461658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461454 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-socket" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:06.461658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.461561 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-credential-socket" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:06.462095 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.462045 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-data" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:06.462195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.462094 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.462195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.462116 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.462195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.462131 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.462195 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.462141 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972de09d-80a9-4a4c-b226-53933aae736f-istiod-ca-cert" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:06.463352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.463328 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/972de09d-80a9-4a4c-b226-53933aae736f-istio-podinfo" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:56:06.463453 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.463406 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-istio-token" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:06.463498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.463467 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-envoy" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:56:06.463539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.463497 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-kube-api-access-n5bl7" pod "972de09d-80a9-4a4c-b226-53933aae736f" (UID: "972de09d-80a9-4a4c-b226-53933aae736f"). InnerVolumeSpecName "kube-api-access-n5bl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:06.563747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.563716 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.563864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.563749 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972de09d-80a9-4a4c-b226-53933aae736f-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.563864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.563762 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972de09d-80a9-4a4c-b226-53933aae736f-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.563864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.563774 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972de09d-80a9-4a4c-b226-53933aae736f-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.563864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.563786 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bl7\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-kube-api-access-n5bl7\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:06.563864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:06.563800 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972de09d-80a9-4a4c-b226-53933aae736f-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:07.170998 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.170975 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 21:56:07.207291 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.207258 5931 generic.go:356] "Generic (PLEG): container finished" podID="972de09d-80a9-4a4c-b226-53933aae736f" containerID="ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0" exitCode=0 Nov 03 21:56:07.207415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.207314 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" Nov 03 21:56:07.207415 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.207332 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" event={"ID":"972de09d-80a9-4a4c-b226-53933aae736f","Type":"ContainerDied","Data":"ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0"} Nov 03 21:56:07.207498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.207414 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm" event={"ID":"972de09d-80a9-4a4c-b226-53933aae736f","Type":"ContainerDied","Data":"c587f0a05fd6be3e365730449bd1011414bf331c55907b85a965a6ea0697c11b"} Nov 03 21:56:07.207498 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.207442 5931 scope.go:122] "RemoveContainer" containerID="ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0" Nov 03 21:56:07.217489 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.217473 5931 scope.go:122] "RemoveContainer" containerID="0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b" Nov 03 21:56:07.225972 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.225956 5931 scope.go:122] "RemoveContainer" containerID="ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0" Nov 03 21:56:07.226213 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:07.226196 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0\": container with ID starting with ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0 not found: ID does not exist" containerID="ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0" Nov 03 21:56:07.226258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.226223 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0"} err="failed to get container status \"ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0\": rpc error: code = NotFound desc = could not find container \"ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0\": container with ID starting with ae39b6002d1f1a21dd232fc6c1c3097a8b3dc5c9d010196788da0c94b393e3a0 not found: ID does not exist" Nov 03 21:56:07.226258 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.226240 5931 scope.go:122] "RemoveContainer" containerID="0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b" Nov 03 21:56:07.226445 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:07.226430 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b\": container with ID starting with 0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b not found: ID does not exist" containerID="0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b" Nov 03 21:56:07.226483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.226456 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b"} err="failed to get container status \"0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b\": rpc error: code = NotFound desc = could not find container \"0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b\": container with ID starting with 0ff803ed28483eb3700fcd7e2811f5b1013e592b004215640fd20b44a7ec6e2b not found: ID does not exist" Nov 03 21:56:07.398147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.398117 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm"] Nov 03 21:56:07.404329 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:07.404305 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-9j6rm"] Nov 03 21:56:08.506113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:08.506083 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972de09d-80a9-4a4c-b226-53933aae736f" path="/var/lib/kubelet/pods/972de09d-80a9-4a4c-b226-53933aae736f/volumes" Nov 03 21:56:10.000978 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:10.000944 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:13.127042 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:13.126968 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:14.278226 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:14.278191 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:15.254821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:15.254784 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:20.077549 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:20.077517 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:20.293145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:20.293114 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:20.510170 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:20.510141 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:25.299878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:25.299842 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:30.924798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:30.924765 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:32.182455 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:32.182400 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:33.068394 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:33.068357 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:56:36.768585 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:36.768532 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:42.205673 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:42.205642 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:46.573215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:46.573162 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:46.784190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:46.784147 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:46.861335 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:46.861304 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5"] Nov 03 21:56:46.861544 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:46.861523 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" podUID="a7875d02-237e-4fd3-a271-801d47d8d62b" containerName="maas-api" containerID="cri-o://d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6" gracePeriod=30 Nov 03 21:56:47.060004 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.059959 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:47.114624 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.114557 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:56:47.237530 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.237494 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls\") pod \"a7875d02-237e-4fd3-a271-801d47d8d62b\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " Nov 03 21:56:47.237726 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.237670 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7875d02-237e-4fd3-a271-801d47d8d62b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7875d02-237e-4fd3-a271-801d47d8d62b-openshift-service-ca\") pod \"a7875d02-237e-4fd3-a271-801d47d8d62b\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " Nov 03 21:56:47.237805 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.237774 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-metrics-certs\") pod \"a7875d02-237e-4fd3-a271-801d47d8d62b\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " Nov 03 21:56:47.237851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.237827 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7875d02-237e-4fd3-a271-801d47d8d62b-kube-api-access-9dvg5\" (UniqueName: \"kubernetes.io/projected/a7875d02-237e-4fd3-a271-801d47d8d62b-kube-api-access-9dvg5\") pod \"a7875d02-237e-4fd3-a271-801d47d8d62b\" (UID: \"a7875d02-237e-4fd3-a271-801d47d8d62b\") " Nov 03 21:56:47.238202 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.238051 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7875d02-237e-4fd3-a271-801d47d8d62b-openshift-service-ca" pod "a7875d02-237e-4fd3-a271-801d47d8d62b" (UID: "a7875d02-237e-4fd3-a271-801d47d8d62b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:47.238294 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.238270 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7875d02-237e-4fd3-a271-801d47d8d62b-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:47.239805 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.239782 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls" pod "a7875d02-237e-4fd3-a271-801d47d8d62b" (UID: "a7875d02-237e-4fd3-a271-801d47d8d62b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:56:47.240145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.240123 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7875d02-237e-4fd3-a271-801d47d8d62b-kube-api-access-9dvg5" pod "a7875d02-237e-4fd3-a271-801d47d8d62b" (UID: "a7875d02-237e-4fd3-a271-801d47d8d62b"). InnerVolumeSpecName "kube-api-access-9dvg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:47.240145 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.240136 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-metrics-certs" pod "a7875d02-237e-4fd3-a271-801d47d8d62b" (UID: "a7875d02-237e-4fd3-a271-801d47d8d62b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:56:47.318842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.318814 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7"] Nov 03 21:56:47.319027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.319010 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" podUID="3b0910c6-14aa-4e2d-b0f1-a08099e558a0" containerName="payload-processing" containerID="cri-o://d7db78142ca1b7b567b50c0e8d638b8f249348974b8eea482dd83eef35d7b933" gracePeriod=30 Nov 03 21:56:47.339671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.339647 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dvg5\" (UniqueName: \"kubernetes.io/projected/a7875d02-237e-4fd3-a271-801d47d8d62b-kube-api-access-9dvg5\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:47.339671 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.339672 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:47.339799 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.339681 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7875d02-237e-4fd3-a271-801d47d8d62b-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:47.375552 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.375498 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7"] Nov 03 21:56:47.375746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.375711 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" podUID="ac23ffb4-19d7-4d35-8344-8c116fb287c7" containerName="payload-pre-processing" containerID="cri-o://d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430" gracePeriod=30 Nov 03 21:56:47.451968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.451941 5931 generic.go:356] "Generic (PLEG): container finished" podID="3b0910c6-14aa-4e2d-b0f1-a08099e558a0" containerID="d7db78142ca1b7b567b50c0e8d638b8f249348974b8eea482dd83eef35d7b933" exitCode=0 Nov 03 21:56:47.452135 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.452016 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" event={"ID":"3b0910c6-14aa-4e2d-b0f1-a08099e558a0","Type":"ContainerDied","Data":"d7db78142ca1b7b567b50c0e8d638b8f249348974b8eea482dd83eef35d7b933"} Nov 03 21:56:47.453341 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.453318 5931 generic.go:356] "Generic (PLEG): container finished" podID="a7875d02-237e-4fd3-a271-801d47d8d62b" containerID="d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6" exitCode=0 Nov 03 21:56:47.453472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.453363 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" Nov 03 21:56:47.453472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.453402 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" event={"ID":"a7875d02-237e-4fd3-a271-801d47d8d62b","Type":"ContainerDied","Data":"d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6"} Nov 03 21:56:47.453472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.453446 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5" event={"ID":"a7875d02-237e-4fd3-a271-801d47d8d62b","Type":"ContainerDied","Data":"1b8ebc810e4ae3e13a2fa95d3aa6fedea763e9eb61977682f89a3a378ccdd231"} Nov 03 21:56:47.453472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.453467 5931 scope.go:122] "RemoveContainer" containerID="d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6" Nov 03 21:56:47.498735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.498708 5931 scope.go:122] "RemoveContainer" containerID="d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6" Nov 03 21:56:47.500386 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:47.500320 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6\": container with ID starting with d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6 not found: ID does not exist" containerID="d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6" Nov 03 21:56:47.500386 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.500356 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6"} err="failed to get container status \"d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6\": rpc error: code = NotFound desc = could not find container \"d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6\": container with ID starting with d8d73f8139961cbc7083c7c3c5b2e7e4efe9e9e36c9364b358bafc0a72b0e1c6 not found: ID does not exist" Nov 03 21:56:47.576418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.576283 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:47.634686 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.633444 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:56:47.661069 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.661031 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5"] Nov 03 21:56:47.664143 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.664110 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-204763-7645bdcb7d-cm4p5"] Nov 03 21:56:47.670795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.670775 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:56:47.744035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.744009 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac23ffb4-19d7-4d35-8344-8c116fb287c7-kube-api-access-fkcqs\" (UniqueName: \"kubernetes.io/projected/ac23ffb4-19d7-4d35-8344-8c116fb287c7-kube-api-access-fkcqs\") pod \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " Nov 03 21:56:47.744126 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.744098 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-kube-api-access-nxnjm\" (UniqueName: \"kubernetes.io/projected/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-kube-api-access-nxnjm\") pod \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " Nov 03 21:56:47.744179 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.744165 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac23ffb4-19d7-4d35-8344-8c116fb287c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac23ffb4-19d7-4d35-8344-8c116fb287c7-plugins-config-volume\") pod \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\" (UID: \"ac23ffb4-19d7-4d35-8344-8c116fb287c7\") " Nov 03 21:56:47.744225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.744209 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-plugins-config-volume\") pod \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\" (UID: \"3b0910c6-14aa-4e2d-b0f1-a08099e558a0\") " Nov 03 21:56:47.744572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.744543 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-plugins-config-volume" pod "3b0910c6-14aa-4e2d-b0f1-a08099e558a0" (UID: "3b0910c6-14aa-4e2d-b0f1-a08099e558a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:47.744572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.744563 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac23ffb4-19d7-4d35-8344-8c116fb287c7-plugins-config-volume" pod "ac23ffb4-19d7-4d35-8344-8c116fb287c7" (UID: "ac23ffb4-19d7-4d35-8344-8c116fb287c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:47.746187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.746170 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac23ffb4-19d7-4d35-8344-8c116fb287c7-kube-api-access-fkcqs" pod "ac23ffb4-19d7-4d35-8344-8c116fb287c7" (UID: "ac23ffb4-19d7-4d35-8344-8c116fb287c7"). InnerVolumeSpecName "kube-api-access-fkcqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:47.746267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.746253 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-kube-api-access-nxnjm" pod "3b0910c6-14aa-4e2d-b0f1-a08099e558a0" (UID: "3b0910c6-14aa-4e2d-b0f1-a08099e558a0"). InnerVolumeSpecName "kube-api-access-nxnjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:47.788834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.788788 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:47.845747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.845726 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkcqs\" (UniqueName: \"kubernetes.io/projected/ac23ffb4-19d7-4d35-8344-8c116fb287c7-kube-api-access-fkcqs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:47.845747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.845748 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnjm\" (UniqueName: \"kubernetes.io/projected/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-kube-api-access-nxnjm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:47.845888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.845757 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac23ffb4-19d7-4d35-8344-8c116fb287c7-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:47.845888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:47.845766 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0910c6-14aa-4e2d-b0f1-a08099e558a0-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:56:48.460593 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.460552 5931 generic.go:356] "Generic (PLEG): container finished" podID="ac23ffb4-19d7-4d35-8344-8c116fb287c7" containerID="d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430" exitCode=0 Nov 03 21:56:48.460791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.460604 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" event={"ID":"ac23ffb4-19d7-4d35-8344-8c116fb287c7","Type":"ContainerDied","Data":"d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430"} Nov 03 21:56:48.460791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.460648 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" Nov 03 21:56:48.460791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.460709 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7" event={"ID":"ac23ffb4-19d7-4d35-8344-8c116fb287c7","Type":"ContainerDied","Data":"ce6a607fe524cd5941055495471414a5f286eb51430147dbd990cf42c07fb68f"} Nov 03 21:56:48.460791 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.460763 5931 scope.go:122] "RemoveContainer" containerID="d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430" Nov 03 21:56:48.462523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.462503 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" Nov 03 21:56:48.462747 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.462537 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7" event={"ID":"3b0910c6-14aa-4e2d-b0f1-a08099e558a0","Type":"ContainerDied","Data":"62e85e0a94bb3231913e5e44faf74e132a5e6313d29a6d14d8669d726cfb743a"} Nov 03 21:56:48.471649 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.471630 5931 scope.go:122] "RemoveContainer" containerID="d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430" Nov 03 21:56:48.471992 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:48.471914 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430\": container with ID starting with d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430 not found: ID does not exist" containerID="d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430" Nov 03 21:56:48.471992 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.471939 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430"} err="failed to get container status \"d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430\": rpc error: code = NotFound desc = could not find container \"d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430\": container with ID starting with d8dad23eefe64ea36ddfdda533b0eb63d719ca971d04e6bc22981587fdd58430 not found: ID does not exist" Nov 03 21:56:48.471992 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.471956 5931 scope.go:122] "RemoveContainer" containerID="d7db78142ca1b7b567b50c0e8d638b8f249348974b8eea482dd83eef35d7b933" Nov 03 21:56:48.492369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492348 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs"] Nov 03 21:56:48.492901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492887 5931 cpu_manager.go:414] "containerMap: removing container" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" Nov 03 21:56:48.492953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492903 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" Nov 03 21:56:48.492953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492921 5931 cpu_manager.go:414] "containerMap: removing container" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" Nov 03 21:56:48.492953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492930 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" Nov 03 21:56:48.492953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492946 5931 cpu_manager.go:414] "containerMap: removing container" podUID="ac23ffb4-19d7-4d35-8344-8c116fb287c7" containerName="payload-pre-processing" Nov 03 21:56:48.492953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492952 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac23ffb4-19d7-4d35-8344-8c116fb287c7" containerName="payload-pre-processing" Nov 03 21:56:48.493103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492978 5931 cpu_manager.go:414] "containerMap: removing container" podUID="a7875d02-237e-4fd3-a271-801d47d8d62b" containerName="maas-api" Nov 03 21:56:48.493103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492983 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7875d02-237e-4fd3-a271-801d47d8d62b" containerName="maas-api" Nov 03 21:56:48.493103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492994 5931 cpu_manager.go:414] "containerMap: removing container" podUID="3b0910c6-14aa-4e2d-b0f1-a08099e558a0" containerName="payload-processing" Nov 03 21:56:48.493103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.492999 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0910c6-14aa-4e2d-b0f1-a08099e558a0" containerName="payload-processing" Nov 03 21:56:48.493103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.493079 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" Nov 03 21:56:48.493103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.493092 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac23ffb4-19d7-4d35-8344-8c116fb287c7" containerName="payload-pre-processing" Nov 03 21:56:48.493103 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.493100 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7875d02-237e-4fd3-a271-801d47d8d62b" containerName="maas-api" Nov 03 21:56:48.493305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.493107 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0910c6-14aa-4e2d-b0f1-a08099e558a0" containerName="payload-processing" Nov 03 21:56:48.497659 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.497643 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.506521 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.506503 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7875d02-237e-4fd3-a271-801d47d8d62b" path="/var/lib/kubelet/pods/a7875d02-237e-4fd3-a271-801d47d8d62b/volumes" Nov 03 21:56:48.549647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.549630 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cd2a9f-serving-cert\"" Nov 03 21:56:48.553345 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.553328 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs"] Nov 03 21:56:48.553755 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.553740 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.553821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.553787 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4npx\" (UniqueName: \"kubernetes.io/projected/a36c20a5-2d81-4faa-926c-65c516796e78-kube-api-access-z4npx\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.553821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.553816 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.553953 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.553929 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36c20a5-2d81-4faa-926c-65c516796e78-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.623165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.623140 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7"] Nov 03 21:56:48.628249 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.628230 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-204763-7f5c447b79-gnqw7"] Nov 03 21:56:48.654719 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.654697 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4npx\" (UniqueName: \"kubernetes.io/projected/a36c20a5-2d81-4faa-926c-65c516796e78-kube-api-access-z4npx\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.654813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.654733 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.654813 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.654753 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36c20a5-2d81-4faa-926c-65c516796e78-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.654930 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:48.654854 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-serving-cert: secret "maas-api-e2e-praxis-b-cd2a9f-serving-cert" not found Nov 03 21:56:48.654930 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.654860 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.654930 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:56:48.654913 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls podName:a36c20a5-2d81-4faa-926c-65c516796e78 nodeName:}" failed. No retries permitted until 2026-11-03 21:56:49.154892324 +0000 UTC m=+2476.765740151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls") pod "maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" (UID: "a36c20a5-2d81-4faa-926c-65c516796e78") : secret "maas-api-e2e-praxis-b-cd2a9f-serving-cert" not found Nov 03 21:56:48.655393 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.655371 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36c20a5-2d81-4faa-926c-65c516796e78-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.657246 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.657225 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.673871 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.673845 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:48.713793 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.713746 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4npx\" (UniqueName: \"kubernetes.io/projected/a36c20a5-2d81-4faa-926c-65c516796e78-kube-api-access-z4npx\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:48.848404 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.848370 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7"] Nov 03 21:56:48.854783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.854743 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-204763-57ff4d46fb-65rn7"] Nov 03 21:56:48.887623 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:48.887583 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:49.126577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:49.126547 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:49.160656 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:49.160604 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:49.163030 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:49.163009 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:49.411816 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:49.411750 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:49.732394 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:49.732334 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:49.785012 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:49.784989 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs"] Nov 03 21:56:49.787652 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:56:49.787596 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36c20a5_2d81_4faa_926c_65c516796e78.slice/crio-3b73f20c38520fa9efaa7a6983ec4146ce3ea14c36d760df47fcc46a14cb439a WatchSource:0}: Error finding container 3b73f20c38520fa9efaa7a6983ec4146ce3ea14c36d760df47fcc46a14cb439a: Status 404 returned error can't find the container with id 3b73f20c38520fa9efaa7a6983ec4146ce3ea14c36d760df47fcc46a14cb439a Nov 03 21:56:49.991674 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:49.991605 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:50.483445 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:50.483411 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" event={"ID":"a36c20a5-2d81-4faa-926c-65c516796e78","Type":"ContainerStarted","Data":"3b73f20c38520fa9efaa7a6983ec4146ce3ea14c36d760df47fcc46a14cb439a"} Nov 03 21:56:50.505085 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:50.505049 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0910c6-14aa-4e2d-b0f1-a08099e558a0" path="/var/lib/kubelet/pods/3b0910c6-14aa-4e2d-b0f1-a08099e558a0/volumes" Nov 03 21:56:50.505561 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:50.505545 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac23ffb4-19d7-4d35-8344-8c116fb287c7" path="/var/lib/kubelet/pods/ac23ffb4-19d7-4d35-8344-8c116fb287c7/volumes" Nov 03 21:56:51.491665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:51.491624 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" event={"ID":"a36c20a5-2d81-4faa-926c-65c516796e78","Type":"ContainerStarted","Data":"163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b"} Nov 03 21:56:51.492029 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:51.491769 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:51.606851 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:51.606792 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" podStartSLOduration=2.7299089050000003 podStartE2EDuration="3.606775278s" totalImagesPullingTime="876.866373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:51.605169222 +0000 UTC m=+2479.216017037" watchObservedRunningTime="2026-11-03 21:56:51.606775278 +0000 UTC m=+2479.217623093" Nov 03 21:56:54.778586 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:54.778554 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:55.603491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:55.603464 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:55.820389 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:55.820359 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:56.040094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:56.040009 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:57.503559 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:57.503530 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 21:56:57.597901 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:57.597868 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:58.053566 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.053534 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792"] Nov 03 21:56:58.054377 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.054351 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="972de09d-80a9-4a4c-b226-53933aae736f" containerName="istio-proxy" Nov 03 21:56:58.057732 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.057716 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.110083 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.110057 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-93c833-openshift-default-dockercfg-bjkl2\"" Nov 03 21:56:58.155454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155423 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52b5bb-445f-410b-a621-9bebdb791b55-istiod-ca-cert\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155454 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155457 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-envoy\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155680 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155486 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-podinfo\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155680 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155596 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-token\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155722 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-socket\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155785 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24s55\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-kube-api-access-24s55\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155812 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155807 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-certs\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155889 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155872 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-data\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.155945 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.155904 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-credential-socket\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.162224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.162126 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792"] Nov 03 21:56:58.219647 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.219602 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:58.257100 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257070 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52b5bb-445f-410b-a621-9bebdb791b55-istiod-ca-cert\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257103 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-envoy\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257132 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-podinfo\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257262 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257152 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-token\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257414 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257336 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-socket\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257468 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257451 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24s55\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-kube-api-access-24s55\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257492 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-certs\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257529 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257521 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-data\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257560 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-credential-socket\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257716 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257665 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-socket\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257880 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52b5bb-445f-410b-a621-9bebdb791b55-istiod-ca-cert\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257887 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-certs\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257922 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-credential-socket\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.257909 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.257960 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-data\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.259587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.259571 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-envoy\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.259739 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.259722 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-podinfo\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.313864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.313806 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24s55\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-kube-api-access-24s55\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.313864 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.313858 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-token\") pod \"e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.371296 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.371267 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:56:58.450232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.450198 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:58.801556 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.801525 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792"] Nov 03 21:56:58.804176 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:56:58.804138 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a52b5bb_445f_410b_a621_9bebdb791b55.slice/crio-4f5d4fdc04cf2ce02acf19ecb2e849f482e81c325fef3012f332fdbae73ae606 WatchSource:0}: Error finding container 4f5d4fdc04cf2ce02acf19ecb2e849f482e81c325fef3012f332fdbae73ae606: Status 404 returned error can't find the container with id 4f5d4fdc04cf2ce02acf19ecb2e849f482e81c325fef3012f332fdbae73ae606 Nov 03 21:56:58.806766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.806734 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:56:58.806834 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.806814 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:56:58.806866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:58.806852 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:56:59.123060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.123028 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t"] Nov 03 21:56:59.128204 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.128180 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.181137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.180955 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6ddbba-serving-cert\"" Nov 03 21:56:59.184176 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.184144 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t"] Nov 03 21:56:59.268666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.268634 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc07e711-7215-44f0-86b5-69a682869a4d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.268827 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.268717 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrstw\" (UniqueName: \"kubernetes.io/projected/cc07e711-7215-44f0-86b5-69a682869a4d-kube-api-access-rrstw\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.268827 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.268785 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.268976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.268956 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.369762 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.369738 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.369883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.369813 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc07e711-7215-44f0-86b5-69a682869a4d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.369926 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.369880 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrstw\" (UniqueName: \"kubernetes.io/projected/cc07e711-7215-44f0-86b5-69a682869a4d-kube-api-access-rrstw\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.369926 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.369908 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.370518 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.370489 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc07e711-7215-44f0-86b5-69a682869a4d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.372032 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.372013 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.372252 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.372234 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.426365 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.426308 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrstw\" (UniqueName: \"kubernetes.io/projected/cc07e711-7215-44f0-86b5-69a682869a4d-kube-api-access-rrstw\") pod \"maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.441358 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.441338 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:56:59.526016 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.525987 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:59.539819 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.539793 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" event={"ID":"2a52b5bb-445f-410b-a621-9bebdb791b55","Type":"ContainerStarted","Data":"e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5"} Nov 03 21:56:59.539923 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.539838 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" event={"ID":"2a52b5bb-445f-410b-a621-9bebdb791b55","Type":"ContainerStarted","Data":"4f5d4fdc04cf2ce02acf19ecb2e849f482e81c325fef3012f332fdbae73ae606"} Nov 03 21:56:59.657830 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.657785 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" podStartSLOduration=1.657772225 podStartE2EDuration="1.657772225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:59.65529282 +0000 UTC m=+2487.266140647" watchObservedRunningTime="2026-11-03 21:56:59.657772225 +0000 UTC m=+2487.268620040" Nov 03 21:56:59.812967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:56:59.812946 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t"] Nov 03 21:56:59.815073 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:56:59.815048 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc07e711_7215_44f0_86b5_69a682869a4d.slice/crio-5ff083f8987a44dbfa4b0029ccac6519c5468979e2b03776a7c03d2f6854cbf8 WatchSource:0}: Error finding container 5ff083f8987a44dbfa4b0029ccac6519c5468979e2b03776a7c03d2f6854cbf8: Status 404 returned error can't find the container with id 5ff083f8987a44dbfa4b0029ccac6519c5468979e2b03776a7c03d2f6854cbf8 Nov 03 21:57:00.371770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:00.371738 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:57:00.376889 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:00.376867 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:57:00.547532 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:00.547506 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" event={"ID":"cc07e711-7215-44f0-86b5-69a682869a4d","Type":"ContainerStarted","Data":"5ff083f8987a44dbfa4b0029ccac6519c5468979e2b03776a7c03d2f6854cbf8"} Nov 03 21:57:00.547770 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:00.547745 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:57:00.548483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:00.548466 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:57:01.554775 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:01.554740 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" event={"ID":"cc07e711-7215-44f0-86b5-69a682869a4d","Type":"ContainerStarted","Data":"6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b"} Nov 03 21:57:01.555144 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:01.554914 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:57:01.675928 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:01.675885 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" podStartSLOduration=1.884699407 podStartE2EDuration="2.675872176s" totalImagesPullingTime="791.172769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:01.672676733 +0000 UTC m=+2489.283524547" watchObservedRunningTime="2026-11-03 21:57:01.675872176 +0000 UTC m=+2489.286719990" Nov 03 21:57:03.067700 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:57:03.067666 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:57:03.320215 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:03.320152 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:03.539781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:03.539755 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:04.877842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:04.877801 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:07.565477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:07.565448 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 21:57:10.189411 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.189380 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:10.419890 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.419857 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:10.509170 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.509093 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc"] Nov 03 21:57:10.513190 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.513154 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.569104 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.569079 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc"] Nov 03 21:57:10.582222 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.582191 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06fe570-6375-402e-9484-119bfea6351e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.582367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.582264 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bmn\" (UniqueName: \"kubernetes.io/projected/f06fe570-6375-402e-9484-119bfea6351e-kube-api-access-f7bmn\") pod \"payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.614867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.614845 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-93c833-dockercfg-mp6dr\"" Nov 03 21:57:10.614867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.614842 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-93c833\"" Nov 03 21:57:10.667352 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.667319 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t"] Nov 03 21:57:10.671562 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.671539 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:10.682920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.682893 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06fe570-6375-402e-9484-119bfea6351e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.683043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.682956 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0f586-7675-46c5-bebb-40b2a291a049-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:10.683043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.683000 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bmn\" (UniqueName: \"kubernetes.io/projected/f06fe570-6375-402e-9484-119bfea6351e-kube-api-access-f7bmn\") pod \"payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.683043 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.683039 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8dm\" (UniqueName: \"kubernetes.io/projected/6ab0f586-7675-46c5-bebb-40b2a291a049-kube-api-access-4z8dm\") pod \"payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:10.683522 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.683503 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06fe570-6375-402e-9484-119bfea6351e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.720441 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.720408 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t"] Nov 03 21:57:10.754462 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.754407 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bmn\" (UniqueName: \"kubernetes.io/projected/f06fe570-6375-402e-9484-119bfea6351e-kube-api-access-f7bmn\") pod \"payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.783862 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.783788 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0f586-7675-46c5-bebb-40b2a291a049-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:10.783862 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.783834 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8dm\" (UniqueName: \"kubernetes.io/projected/6ab0f586-7675-46c5-bebb-40b2a291a049-kube-api-access-4z8dm\") pod \"payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:10.784477 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.784454 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0f586-7675-46c5-bebb-40b2a291a049-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:10.847315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.847286 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8dm\" (UniqueName: \"kubernetes.io/projected/6ab0f586-7675-46c5-bebb-40b2a291a049-kube-api-access-4z8dm\") pod \"payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:10.847680 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.847658 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:10.985351 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:10.985320 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:11.247517 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.247475 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc"] Nov 03 21:57:11.251758 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:57:11.251731 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06fe570_6375_402e_9484_119bfea6351e.slice/crio-fd230a5d8dbeb967c0919f0f67e9e09a54d8fbe092301502de3e8cac2de290b3 WatchSource:0}: Error finding container fd230a5d8dbeb967c0919f0f67e9e09a54d8fbe092301502de3e8cac2de290b3: Status 404 returned error can't find the container with id fd230a5d8dbeb967c0919f0f67e9e09a54d8fbe092301502de3e8cac2de290b3 Nov 03 21:57:11.381180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.380731 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t"] Nov 03 21:57:11.381519 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:57:11.381496 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab0f586_7675_46c5_bebb_40b2a291a049.slice/crio-2169becaf37dd708fc627fb8d4896e197a400cce5ef285cc30146728ce6db6da WatchSource:0}: Error finding container 2169becaf37dd708fc627fb8d4896e197a400cce5ef285cc30146728ce6db6da: Status 404 returned error can't find the container with id 2169becaf37dd708fc627fb8d4896e197a400cce5ef285cc30146728ce6db6da Nov 03 21:57:11.618686 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.618650 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" event={"ID":"f06fe570-6375-402e-9484-119bfea6351e","Type":"ContainerStarted","Data":"253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249"} Nov 03 21:57:11.618883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.618734 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:11.618883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.618759 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" event={"ID":"f06fe570-6375-402e-9484-119bfea6351e","Type":"ContainerStarted","Data":"fd230a5d8dbeb967c0919f0f67e9e09a54d8fbe092301502de3e8cac2de290b3"} Nov 03 21:57:11.620115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.620090 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" event={"ID":"6ab0f586-7675-46c5-bebb-40b2a291a049","Type":"ContainerStarted","Data":"95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196"} Nov 03 21:57:11.620284 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.620268 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" event={"ID":"6ab0f586-7675-46c5-bebb-40b2a291a049","Type":"ContainerStarted","Data":"2169becaf37dd708fc627fb8d4896e197a400cce5ef285cc30146728ce6db6da"} Nov 03 21:57:11.620399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.620387 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:11.852648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.851000 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" podStartSLOduration=1.850981187 podStartE2EDuration="1.850981187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:11.735846681 +0000 UTC m=+2499.346694498" watchObservedRunningTime="2026-11-03 21:57:11.850981187 +0000 UTC m=+2499.461829004" Nov 03 21:57:11.852648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:11.851632 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" podStartSLOduration=1.851604448 podStartE2EDuration="1.851604448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:11.849978637 +0000 UTC m=+2499.460826453" watchObservedRunningTime="2026-11-03 21:57:11.851604448 +0000 UTC m=+2499.462452264" Nov 03 21:57:15.630324 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:15.630287 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:16.747244 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:16.747210 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:17.686754 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:17.686728 5931 generic.go:356] "Generic (PLEG): container finished" podID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerID="6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab" exitCode=0 Nov 03 21:57:17.686904 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:17.686797 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" event={"ID":"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791","Type":"ContainerDied","Data":"6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab"} Nov 03 21:57:17.687232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:17.687214 5931 scope.go:122] "RemoveContainer" containerID="6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab" Nov 03 21:57:17.688796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:17.688767 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:57:17.688867 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:17.688840 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:57:17.688908 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:17.688870 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:57:17.931232 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:17.931157 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:18.696029 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:18.695995 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" event={"ID":"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791","Type":"ContainerStarted","Data":"99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1"} Nov 03 21:57:18.756592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:18.756544 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:18.969667 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:18.969575 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:19.104706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:19.104674 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:57:19.105980 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:19.105945 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:57:19.106127 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:19.106006 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:57:19.987392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:19.987357 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:57:20.104976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:20.104945 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:57:20.105156 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:20.104999 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:57:20.410173 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:20.410146 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:21.104325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:21.104298 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:57:21.104784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:21.104761 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:57:22.047471 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:22.047438 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:22.104224 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:22.104196 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:57:22.104349 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:22.104252 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:57:22.358129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:22.358104 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:22.621040 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:22.620945 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:22.654491 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:22.654457 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 21:57:22.654665 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:22.654533 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 21:57:23.048118 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:23.048032 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:23.104920 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:23.104888 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:57:23.105261 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:23.104930 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 21:57:23.276589 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:23.276551 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:23.492302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:23.492271 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:24.108839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:24.108813 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:57:24.109790 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:24.109762 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 21:57:28.047115 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:28.047084 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:28.799297 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:28.799261 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:29.020602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:29.020564 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:29.298625 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:29.298581 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:33.070762 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:57:33.070729 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:57:33.222933 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:33.222902 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:37.831544 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:37.831506 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:38.046426 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.046393 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:38.305681 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.305645 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:38.333137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.333109 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg"] Nov 03 21:57:38.338497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.338480 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.394900 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.394866 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-95d604\"" Nov 03 21:57:38.395021 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.394895 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-95d604-dockercfg-t49mb\"" Nov 03 21:57:38.396267 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.396246 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg"] Nov 03 21:57:38.475279 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.475253 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5"] Nov 03 21:57:38.479257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.479241 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:38.492351 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.492330 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqrd\" (UniqueName: \"kubernetes.io/projected/1482d9f0-ed14-4549-80ce-4d04364ab142-kube-api-access-jvqrd\") pod \"payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.492557 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.492534 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1482d9f0-ed14-4549-80ce-4d04364ab142-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.564182 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.564100 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:38.566510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.566482 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5"] Nov 03 21:57:38.594286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.594256 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxst\" (UniqueName: \"kubernetes.io/projected/0d460f72-809b-4ba4-a50e-b667d1b29dce-kube-api-access-xsxst\") pod \"payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:38.594451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.594338 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1482d9f0-ed14-4549-80ce-4d04364ab142-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.594451 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.594404 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqrd\" (UniqueName: \"kubernetes.io/projected/1482d9f0-ed14-4549-80ce-4d04364ab142-kube-api-access-jvqrd\") pod \"payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.594576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.594456 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d460f72-809b-4ba4-a50e-b667d1b29dce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:38.595301 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.595273 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1482d9f0-ed14-4549-80ce-4d04364ab142-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.650976 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.650953 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqrd\" (UniqueName: \"kubernetes.io/projected/1482d9f0-ed14-4549-80ce-4d04364ab142-kube-api-access-jvqrd\") pod \"payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.651727 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.651709 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:38.695820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.695789 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxst\" (UniqueName: \"kubernetes.io/projected/0d460f72-809b-4ba4-a50e-b667d1b29dce-kube-api-access-xsxst\") pod \"payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:38.695957 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.695890 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d460f72-809b-4ba4-a50e-b667d1b29dce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:38.696513 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.696493 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d460f72-809b-4ba4-a50e-b667d1b29dce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:38.752025 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.751986 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxst\" (UniqueName: \"kubernetes.io/projected/0d460f72-809b-4ba4-a50e-b667d1b29dce-kube-api-access-xsxst\") pod \"payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:38.792768 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:38.792740 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:39.025286 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.025248 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg"] Nov 03 21:57:39.027654 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:57:39.027599 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1482d9f0_ed14_4549_80ce_4d04364ab142.slice/crio-fd0874c64114bc893427c3b109deb667f69dc16099484e1f9b37c607c5a4ccde WatchSource:0}: Error finding container fd0874c64114bc893427c3b109deb667f69dc16099484e1f9b37c607c5a4ccde: Status 404 returned error can't find the container with id fd0874c64114bc893427c3b109deb667f69dc16099484e1f9b37c607c5a4ccde Nov 03 21:57:39.172181 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.172158 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5"] Nov 03 21:57:39.173898 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:57:39.173873 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d460f72_809b_4ba4_a50e_b667d1b29dce.slice/crio-e1069695f6d4c0958ac3e306f23ec48299345fd82e5a2f1261f9a18fa7ef396e WatchSource:0}: Error finding container e1069695f6d4c0958ac3e306f23ec48299345fd82e5a2f1261f9a18fa7ef396e: Status 404 returned error can't find the container with id e1069695f6d4c0958ac3e306f23ec48299345fd82e5a2f1261f9a18fa7ef396e Nov 03 21:57:39.848338 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.848303 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" event={"ID":"1482d9f0-ed14-4549-80ce-4d04364ab142","Type":"ContainerStarted","Data":"f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6"} Nov 03 21:57:39.848508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.848358 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" event={"ID":"1482d9f0-ed14-4549-80ce-4d04364ab142","Type":"ContainerStarted","Data":"fd0874c64114bc893427c3b109deb667f69dc16099484e1f9b37c607c5a4ccde"} Nov 03 21:57:39.848508 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.848383 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:39.849852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.849828 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" event={"ID":"0d460f72-809b-4ba4-a50e-b667d1b29dce","Type":"ContainerStarted","Data":"ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501"} Nov 03 21:57:39.849958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.849872 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:39.849958 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.849896 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" event={"ID":"0d460f72-809b-4ba4-a50e-b667d1b29dce","Type":"ContainerStarted","Data":"e1069695f6d4c0958ac3e306f23ec48299345fd82e5a2f1261f9a18fa7ef396e"} Nov 03 21:57:39.961497 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:39.961453 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" podStartSLOduration=1.961441426 podStartE2EDuration="1.961441426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:39.959723664 +0000 UTC m=+2527.570587436" watchObservedRunningTime="2026-11-03 21:57:39.961441426 +0000 UTC m=+2527.572289240" Nov 03 21:57:40.074603 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:40.073635 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" podStartSLOduration=2.073605275 podStartE2EDuration="2.073605275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:40.070715929 +0000 UTC m=+2527.681563744" watchObservedRunningTime="2026-11-03 21:57:40.073605275 +0000 UTC m=+2527.684453116" Nov 03 21:57:43.673357 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:43.673281 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:43.883691 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:43.883660 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s"] Nov 03 21:57:43.992662 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:43.992587 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv"] Nov 03 21:57:44.337406 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:44.337317 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:44.546863 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:44.546829 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:48.371903 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:48.371869 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:48.884549 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:48.884515 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" podUID="ba3bdfed-684a-49a2-9978-79af19f96c24" containerName="payload-processing" containerID="cri-o://a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104" gracePeriod=25 Nov 03 21:57:48.993303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:48.993276 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" podUID="c819184f-9e18-43c1-a6a8-8633597ab352" containerName="payload-pre-processing" containerID="cri-o://f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c" gracePeriod=25 Nov 03 21:57:49.161703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.161682 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:57:49.193097 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.193058 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3bdfed-684a-49a2-9978-79af19f96c24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3bdfed-684a-49a2-9978-79af19f96c24-plugins-config-volume\") pod \"ba3bdfed-684a-49a2-9978-79af19f96c24\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " Nov 03 21:57:49.193241 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.193226 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3bdfed-684a-49a2-9978-79af19f96c24-kube-api-access-jg8nb\" (UniqueName: \"kubernetes.io/projected/ba3bdfed-684a-49a2-9978-79af19f96c24-kube-api-access-jg8nb\") pod \"ba3bdfed-684a-49a2-9978-79af19f96c24\" (UID: \"ba3bdfed-684a-49a2-9978-79af19f96c24\") " Nov 03 21:57:49.193480 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.193440 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3bdfed-684a-49a2-9978-79af19f96c24-plugins-config-volume" pod "ba3bdfed-684a-49a2-9978-79af19f96c24" (UID: "ba3bdfed-684a-49a2-9978-79af19f96c24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:57:49.193741 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.193726 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3bdfed-684a-49a2-9978-79af19f96c24-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:57:49.195538 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.195516 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3bdfed-684a-49a2-9978-79af19f96c24-kube-api-access-jg8nb" pod "ba3bdfed-684a-49a2-9978-79af19f96c24" (UID: "ba3bdfed-684a-49a2-9978-79af19f96c24"). InnerVolumeSpecName "kube-api-access-jg8nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:49.263510 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.263487 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:57:49.295025 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.294992 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c819184f-9e18-43c1-a6a8-8633597ab352-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c819184f-9e18-43c1-a6a8-8633597ab352-plugins-config-volume\") pod \"c819184f-9e18-43c1-a6a8-8633597ab352\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " Nov 03 21:57:49.295157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.295135 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c819184f-9e18-43c1-a6a8-8633597ab352-kube-api-access-x72z9\" (UniqueName: \"kubernetes.io/projected/c819184f-9e18-43c1-a6a8-8633597ab352-kube-api-access-x72z9\") pod \"c819184f-9e18-43c1-a6a8-8633597ab352\" (UID: \"c819184f-9e18-43c1-a6a8-8633597ab352\") " Nov 03 21:57:49.295360 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.295336 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c819184f-9e18-43c1-a6a8-8633597ab352-plugins-config-volume" pod "c819184f-9e18-43c1-a6a8-8633597ab352" (UID: "c819184f-9e18-43c1-a6a8-8633597ab352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:57:49.295582 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.295557 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8nb\" (UniqueName: \"kubernetes.io/projected/ba3bdfed-684a-49a2-9978-79af19f96c24-kube-api-access-jg8nb\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:57:49.295582 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.295581 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c819184f-9e18-43c1-a6a8-8633597ab352-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:57:49.297087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.297066 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c819184f-9e18-43c1-a6a8-8633597ab352-kube-api-access-x72z9" pod "c819184f-9e18-43c1-a6a8-8633597ab352" (UID: "c819184f-9e18-43c1-a6a8-8633597ab352"). InnerVolumeSpecName "kube-api-access-x72z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:49.397014 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.396952 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72z9\" (UniqueName: \"kubernetes.io/projected/c819184f-9e18-43c1-a6a8-8633597ab352-kube-api-access-x72z9\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:57:49.915165 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.915134 5931 generic.go:356] "Generic (PLEG): container finished" podID="c819184f-9e18-43c1-a6a8-8633597ab352" containerID="f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c" exitCode=0 Nov 03 21:57:49.915309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.915188 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" Nov 03 21:57:49.915309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.915214 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" event={"ID":"c819184f-9e18-43c1-a6a8-8633597ab352","Type":"ContainerDied","Data":"f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c"} Nov 03 21:57:49.915309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.915282 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv" event={"ID":"c819184f-9e18-43c1-a6a8-8633597ab352","Type":"ContainerDied","Data":"38afded70373e50bb33da5d4ba8d49f48a416d93c0a7ab9004090e9f8086282f"} Nov 03 21:57:49.915309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.915309 5931 scope.go:122] "RemoveContainer" containerID="f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c" Nov 03 21:57:49.916758 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.916727 5931 generic.go:356] "Generic (PLEG): container finished" podID="ba3bdfed-684a-49a2-9978-79af19f96c24" containerID="a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104" exitCode=0 Nov 03 21:57:49.916852 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.916812 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" event={"ID":"ba3bdfed-684a-49a2-9978-79af19f96c24","Type":"ContainerDied","Data":"a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104"} Nov 03 21:57:49.916902 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.916857 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" event={"ID":"ba3bdfed-684a-49a2-9978-79af19f96c24","Type":"ContainerDied","Data":"91225cd1cbf5249956d674fe486a8f5be57687339527509e49ba9e5b1e9ed0e4"} Nov 03 21:57:49.916986 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.916949 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s" Nov 03 21:57:49.926820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.926800 5931 scope.go:122] "RemoveContainer" containerID="f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c" Nov 03 21:57:49.927070 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:57:49.927053 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c\": container with ID starting with f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c not found: ID does not exist" containerID="f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c" Nov 03 21:57:49.927137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.927082 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c"} err="failed to get container status \"f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c\": rpc error: code = NotFound desc = could not find container \"f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c\": container with ID starting with f3d2d15f8e79c1b6b877aac0ac85753365811f718d5ce343638685c3886c8a9c not found: ID does not exist" Nov 03 21:57:49.927137 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.927106 5931 scope.go:122] "RemoveContainer" containerID="a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104" Nov 03 21:57:49.935540 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.935524 5931 scope.go:122] "RemoveContainer" containerID="a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104" Nov 03 21:57:49.935779 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:57:49.935763 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104\": container with ID starting with a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104 not found: ID does not exist" containerID="a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104" Nov 03 21:57:49.935839 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:49.935785 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104"} err="failed to get container status \"a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104\": rpc error: code = NotFound desc = could not find container \"a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104\": container with ID starting with a5bdb64ef03591c18a7268af3c24436c1256f55899883522237e9a6dbe05d104 not found: ID does not exist" Nov 03 21:57:50.077878 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.077855 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv"] Nov 03 21:57:50.127257 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.127233 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-292cf1-85574cdb7f-kndkv"] Nov 03 21:57:50.186648 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.186586 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s"] Nov 03 21:57:50.189807 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.189785 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-292cf1-749b6d8b96-w458s"] Nov 03 21:57:50.202938 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.202919 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:50.506305 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.506246 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3bdfed-684a-49a2-9978-79af19f96c24" path="/var/lib/kubelet/pods/ba3bdfed-684a-49a2-9978-79af19f96c24/volumes" Nov 03 21:57:50.506674 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.506659 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c819184f-9e18-43c1-a6a8-8633597ab352" path="/var/lib/kubelet/pods/c819184f-9e18-43c1-a6a8-8633597ab352/volumes" Nov 03 21:57:50.858550 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.858526 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 21:57:50.858699 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:50.858583 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 21:57:53.169147 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:53.169114 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:56.689682 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:56.689644 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:58.307187 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:57:58.307157 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:00.428783 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:00.428736 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:01.189198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:01.189168 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68"] Nov 03 21:58:01.311705 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:01.311663 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:02.563539 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:02.563506 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:03.087851 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:03.087811 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:58:03.199640 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:03.199582 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:03.466954 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:03.466929 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:03.696045 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:03.696013 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:06.997452 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:06.997424 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:07.291932 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:07.291855 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:08.605469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:08.605437 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:08.854881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:08.854846 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:09.201850 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:09.201818 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:11.962355 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:11.962322 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:12.177560 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:12.177477 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:14.149523 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:14.149492 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:15.366421 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:15.366393 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f"] Nov 03 21:58:15.473981 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:15.473948 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq"] Nov 03 21:58:15.843587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:15.843554 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:16.066973 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.066947 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:16.209757 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.209680 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" podUID="0ec72e18-acc9-46dc-9d32-69bda7d11a18" containerName="main" containerID="cri-o://efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac" gracePeriod=45 Nov 03 21:58:16.265777 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.265743 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" podUID="0ec72e18-acc9-46dc-9d32-69bda7d11a18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.136:8000/ready\": dial tcp 10.134.0.136:8000: connect: connection refused" Nov 03 21:58:16.478022 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.477969 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:58:16.654821 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.654794 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec72e18-acc9-46dc-9d32-69bda7d11a18-kube-api-access-69mfc\" (UniqueName: \"kubernetes.io/projected/0ec72e18-acc9-46dc-9d32-69bda7d11a18-kube-api-access-69mfc\") pod \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " Nov 03 21:58:16.654967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.654872 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-home\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-home\") pod \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " Nov 03 21:58:16.654967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.654899 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-model-cache\") pod \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " Nov 03 21:58:16.655060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.654964 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tls-certs\") pod \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " Nov 03 21:58:16.655060 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655025 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tmp-dir\") pod \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " Nov 03 21:58:16.655136 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655075 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-dshm\") pod \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\" (UID: \"0ec72e18-acc9-46dc-9d32-69bda7d11a18\") " Nov 03 21:58:16.655198 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655149 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-home" pod "0ec72e18-acc9-46dc-9d32-69bda7d11a18" (UID: "0ec72e18-acc9-46dc-9d32-69bda7d11a18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:58:16.655269 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655228 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-model-cache" pod "0ec72e18-acc9-46dc-9d32-69bda7d11a18" (UID: "0ec72e18-acc9-46dc-9d32-69bda7d11a18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:58:16.655330 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655289 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tmp-dir" pod "0ec72e18-acc9-46dc-9d32-69bda7d11a18" (UID: "0ec72e18-acc9-46dc-9d32-69bda7d11a18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:58:16.655605 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655584 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:16.655743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655628 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:16.655743 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.655643 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:16.657113 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.657092 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec72e18-acc9-46dc-9d32-69bda7d11a18-kube-api-access-69mfc" pod "0ec72e18-acc9-46dc-9d32-69bda7d11a18" (UID: "0ec72e18-acc9-46dc-9d32-69bda7d11a18"). InnerVolumeSpecName "kube-api-access-69mfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:58:16.657706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.657686 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tls-certs" pod "0ec72e18-acc9-46dc-9d32-69bda7d11a18" (UID: "0ec72e18-acc9-46dc-9d32-69bda7d11a18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:58:16.657706 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.657693 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-dshm" pod "0ec72e18-acc9-46dc-9d32-69bda7d11a18" (UID: "0ec72e18-acc9-46dc-9d32-69bda7d11a18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:58:16.757024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.756971 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec72e18-acc9-46dc-9d32-69bda7d11a18-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:16.757024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.756994 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec72e18-acc9-46dc-9d32-69bda7d11a18-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:16.757024 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:16.757004 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69mfc\" (UniqueName: \"kubernetes.io/projected/0ec72e18-acc9-46dc-9d32-69bda7d11a18-kube-api-access-69mfc\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:17.083356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.083277 5931 generic.go:356] "Generic (PLEG): container finished" podID="0ec72e18-acc9-46dc-9d32-69bda7d11a18" containerID="efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac" exitCode=0 Nov 03 21:58:17.083356 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.083325 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" Nov 03 21:58:17.083543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.083356 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" event={"ID":"0ec72e18-acc9-46dc-9d32-69bda7d11a18","Type":"ContainerDied","Data":"efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac"} Nov 03 21:58:17.083543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.083399 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68" event={"ID":"0ec72e18-acc9-46dc-9d32-69bda7d11a18","Type":"ContainerDied","Data":"c985cabb06cb4cac1f9651a0391084a27dd11d804e61f05a20a42dd076e41a19"} Nov 03 21:58:17.083543 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.083420 5931 scope.go:122] "RemoveContainer" containerID="efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac" Nov 03 21:58:17.093968 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.093951 5931 scope.go:122] "RemoveContainer" containerID="efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac" Nov 03 21:58:17.094232 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:17.094215 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac\": container with ID starting with efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac not found: ID does not exist" containerID="efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac" Nov 03 21:58:17.094303 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.094237 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac"} err="failed to get container status \"efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac\": rpc error: code = NotFound desc = could not find container \"efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac\": container with ID starting with efa7febbb555f1cc130777f5a72e946b235c0fa8ccec4159510fc8aed69c23ac not found: ID does not exist" Nov 03 21:58:17.245751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.245719 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68"] Nov 03 21:58:17.250399 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:17.250365 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/tenantcfg-model-6ddbba-kserve-c7f5457c7-tfn68"] Nov 03 21:58:18.505230 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:18.505197 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec72e18-acc9-46dc-9d32-69bda7d11a18" path="/var/lib/kubelet/pods/0ec72e18-acc9-46dc-9d32-69bda7d11a18/volumes" Nov 03 21:58:19.507905 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:19.507879 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:20.367063 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.367028 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" podUID="a6839d4a-43e3-4c84-b3a4-30ca5b38283d" containerName="payload-processing" containerID="cri-o://304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8" gracePeriod=25 Nov 03 21:58:20.474214 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.474168 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" podUID="fddc1020-8706-4143-9035-8e50bce88add" containerName="payload-pre-processing" containerID="cri-o://eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad" gracePeriod=25 Nov 03 21:58:20.615798 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.615770 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:58:20.706766 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.706739 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-plugins-config-volume\") pod \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " Nov 03 21:58:20.706914 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.706884 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-kube-api-access-wd29j\" (UniqueName: \"kubernetes.io/projected/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-kube-api-access-wd29j\") pod \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\" (UID: \"a6839d4a-43e3-4c84-b3a4-30ca5b38283d\") " Nov 03 21:58:20.707154 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.707121 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-plugins-config-volume" pod "a6839d4a-43e3-4c84-b3a4-30ca5b38283d" (UID: "a6839d4a-43e3-4c84-b3a4-30ca5b38283d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:58:20.707346 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.707328 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:20.709362 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.709345 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-kube-api-access-wd29j" pod "a6839d4a-43e3-4c84-b3a4-30ca5b38283d" (UID: "a6839d4a-43e3-4c84-b3a4-30ca5b38283d"). InnerVolumeSpecName "kube-api-access-wd29j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:58:20.720685 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.720665 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:58:20.807966 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.807938 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddc1020-8706-4143-9035-8e50bce88add-kube-api-access-vg6sx\" (UniqueName: \"kubernetes.io/projected/fddc1020-8706-4143-9035-8e50bce88add-kube-api-access-vg6sx\") pod \"fddc1020-8706-4143-9035-8e50bce88add\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " Nov 03 21:58:20.808072 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.807990 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fddc1020-8706-4143-9035-8e50bce88add-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc1020-8706-4143-9035-8e50bce88add-plugins-config-volume\") pod \"fddc1020-8706-4143-9035-8e50bce88add\" (UID: \"fddc1020-8706-4143-9035-8e50bce88add\") " Nov 03 21:58:20.808263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.808243 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd29j\" (UniqueName: \"kubernetes.io/projected/a6839d4a-43e3-4c84-b3a4-30ca5b38283d-kube-api-access-wd29j\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:20.808354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.808323 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fddc1020-8706-4143-9035-8e50bce88add-plugins-config-volume" pod "fddc1020-8706-4143-9035-8e50bce88add" (UID: "fddc1020-8706-4143-9035-8e50bce88add"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:58:20.809952 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.809931 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddc1020-8706-4143-9035-8e50bce88add-kube-api-access-vg6sx" pod "fddc1020-8706-4143-9035-8e50bce88add" (UID: "fddc1020-8706-4143-9035-8e50bce88add"). InnerVolumeSpecName "kube-api-access-vg6sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:58:20.909037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.908974 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc1020-8706-4143-9035-8e50bce88add-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:20.909037 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:20.908997 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6sx\" (UniqueName: \"kubernetes.io/projected/fddc1020-8706-4143-9035-8e50bce88add-kube-api-access-vg6sx\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:21.111801 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.111770 5931 generic.go:356] "Generic (PLEG): container finished" podID="fddc1020-8706-4143-9035-8e50bce88add" containerID="eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad" exitCode=0 Nov 03 21:58:21.111971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.111809 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" Nov 03 21:58:21.111971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.111855 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" event={"ID":"fddc1020-8706-4143-9035-8e50bce88add","Type":"ContainerDied","Data":"eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad"} Nov 03 21:58:21.111971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.111917 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq" event={"ID":"fddc1020-8706-4143-9035-8e50bce88add","Type":"ContainerDied","Data":"0c32526d944d61da81ad28e32b6a888e0b2fdb30ac50831996eba4d8bee7c068"} Nov 03 21:58:21.111971 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.111947 5931 scope.go:122] "RemoveContainer" containerID="eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad" Nov 03 21:58:21.115784 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.115745 5931 generic.go:356] "Generic (PLEG): container finished" podID="a6839d4a-43e3-4c84-b3a4-30ca5b38283d" containerID="304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8" exitCode=0 Nov 03 21:58:21.115893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.115785 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" event={"ID":"a6839d4a-43e3-4c84-b3a4-30ca5b38283d","Type":"ContainerDied","Data":"304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8"} Nov 03 21:58:21.115893 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.115868 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" event={"ID":"a6839d4a-43e3-4c84-b3a4-30ca5b38283d","Type":"ContainerDied","Data":"a86fc84e377c209705818c2dc59eda296830b17dc2b9d154a867f1364c3ed4ae"} Nov 03 21:58:21.116009 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.115961 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f" Nov 03 21:58:21.131778 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.131760 5931 scope.go:122] "RemoveContainer" containerID="eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad" Nov 03 21:58:21.132033 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:21.132017 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad\": container with ID starting with eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad not found: ID does not exist" containerID="eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad" Nov 03 21:58:21.132080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.132039 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad"} err="failed to get container status \"eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad\": rpc error: code = NotFound desc = could not find container \"eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad\": container with ID starting with eea35c4efb75b673594632aeef314775a7edf28565bcb282ae08242156c78aad not found: ID does not exist" Nov 03 21:58:21.132080 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.132053 5931 scope.go:122] "RemoveContainer" containerID="304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8" Nov 03 21:58:21.140630 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.140589 5931 scope.go:122] "RemoveContainer" containerID="304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8" Nov 03 21:58:21.140886 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:21.140867 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8\": container with ID starting with 304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8 not found: ID does not exist" containerID="304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8" Nov 03 21:58:21.140941 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.140892 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8"} err="failed to get container status \"304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8\": rpc error: code = NotFound desc = could not find container \"304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8\": container with ID starting with 304fc10b05619002674d439f6628de0fa933557cacc2d6ce3614fe9d67b2bce8 not found: ID does not exist" Nov 03 21:58:21.274177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.274127 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq"] Nov 03 21:58:21.323157 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.323131 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c993b8-c776bc4bc-hkssq"] Nov 03 21:58:21.382317 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.382299 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f"] Nov 03 21:58:21.388086 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:21.388067 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c993b8-d656b4c96-n7s9f"] Nov 03 21:58:22.505238 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:22.505208 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6839d4a-43e3-4c84-b3a4-30ca5b38283d" path="/var/lib/kubelet/pods/a6839d4a-43e3-4c84-b3a4-30ca5b38283d/volumes" Nov 03 21:58:22.505592 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:22.505565 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fddc1020-8706-4143-9035-8e50bce88add" path="/var/lib/kubelet/pods/fddc1020-8706-4143-9035-8e50bce88add/volumes" Nov 03 21:58:24.944929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:24.944897 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:29.868856 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.868823 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6"] Nov 03 21:58:29.869328 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869315 5931 cpu_manager.go:414] "containerMap: removing container" podUID="ba3bdfed-684a-49a2-9978-79af19f96c24" containerName="payload-processing" Nov 03 21:58:29.869369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869329 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3bdfed-684a-49a2-9978-79af19f96c24" containerName="payload-processing" Nov 03 21:58:29.869369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869343 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0ec72e18-acc9-46dc-9d32-69bda7d11a18" containerName="main" Nov 03 21:58:29.869369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869348 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec72e18-acc9-46dc-9d32-69bda7d11a18" containerName="main" Nov 03 21:58:29.869369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869362 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c819184f-9e18-43c1-a6a8-8633597ab352" containerName="payload-pre-processing" Nov 03 21:58:29.869369 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869367 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c819184f-9e18-43c1-a6a8-8633597ab352" containerName="payload-pre-processing" Nov 03 21:58:29.869515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869384 5931 cpu_manager.go:414] "containerMap: removing container" podUID="fddc1020-8706-4143-9035-8e50bce88add" containerName="payload-pre-processing" Nov 03 21:58:29.869515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869389 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fddc1020-8706-4143-9035-8e50bce88add" containerName="payload-pre-processing" Nov 03 21:58:29.869515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869404 5931 cpu_manager.go:414] "containerMap: removing container" podUID="a6839d4a-43e3-4c84-b3a4-30ca5b38283d" containerName="payload-processing" Nov 03 21:58:29.869515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869409 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6839d4a-43e3-4c84-b3a4-30ca5b38283d" containerName="payload-processing" Nov 03 21:58:29.869515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869495 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3bdfed-684a-49a2-9978-79af19f96c24" containerName="payload-processing" Nov 03 21:58:29.869515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869503 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec72e18-acc9-46dc-9d32-69bda7d11a18" containerName="main" Nov 03 21:58:29.869515 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869510 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c819184f-9e18-43c1-a6a8-8633597ab352" containerName="payload-pre-processing" Nov 03 21:58:29.869795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869519 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fddc1020-8706-4143-9035-8e50bce88add" containerName="payload-pre-processing" Nov 03 21:58:29.869795 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.869526 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6839d4a-43e3-4c84-b3a4-30ca5b38283d" containerName="payload-processing" Nov 03 21:58:29.872866 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.872852 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:29.926398 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.926371 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-95d604-serving-cert\"" Nov 03 21:58:29.978314 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.978282 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6"] Nov 03 21:58:29.990711 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.990683 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa25de6-4fd2-4a96-9205-32d2d02b5642-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:29.990814 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.990753 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9mqc\" (UniqueName: \"kubernetes.io/projected/6fa25de6-4fd2-4a96-9205-32d2d02b5642-kube-api-access-v9mqc\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:29.990853 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.990833 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:29.990913 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:29.990879 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-metrics-certs\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.092094 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.092067 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9mqc\" (UniqueName: \"kubernetes.io/projected/6fa25de6-4fd2-4a96-9205-32d2d02b5642-kube-api-access-v9mqc\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.092239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.092118 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.092239 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.092164 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-metrics-certs\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.092392 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:30.092278 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-serving-cert: secret "maas-api-e2e-praxis-b-95d604-serving-cert" not found Nov 03 21:58:30.092392 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.092296 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa25de6-4fd2-4a96-9205-32d2d02b5642-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.092392 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:30.092357 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls podName:6fa25de6-4fd2-4a96-9205-32d2d02b5642 nodeName:}" failed. No retries permitted until 2026-11-03 21:58:30.592336143 +0000 UTC m=+2578.203183959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls") pod "maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" (UID: "6fa25de6-4fd2-4a96-9205-32d2d02b5642") : secret "maas-api-e2e-praxis-b-95d604-serving-cert" not found Nov 03 21:58:30.092894 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.092876 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa25de6-4fd2-4a96-9205-32d2d02b5642-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.094522 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.094502 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-metrics-certs\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.148995 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.148941 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9mqc\" (UniqueName: \"kubernetes.io/projected/6fa25de6-4fd2-4a96-9205-32d2d02b5642-kube-api-access-v9mqc\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.188208 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.188182 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:30.414873 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.414793 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:30.598253 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.598228 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.600602 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.600570 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls\") pod \"maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:30.786108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:30.786043 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:31.161325 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:31.161298 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6"] Nov 03 21:58:31.163440 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:58:31.163414 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa25de6_4fd2_4a96_9205_32d2d02b5642.slice/crio-98b31cbf59756172f0e87e741f7928fff3c39ba77b99e7d1f2923c98c7bee052 WatchSource:0}: Error finding container 98b31cbf59756172f0e87e741f7928fff3c39ba77b99e7d1f2923c98c7bee052: Status 404 returned error can't find the container with id 98b31cbf59756172f0e87e741f7928fff3c39ba77b99e7d1f2923c98c7bee052 Nov 03 21:58:31.185929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:31.185904 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" event={"ID":"6fa25de6-4fd2-4a96-9205-32d2d02b5642","Type":"ContainerStarted","Data":"98b31cbf59756172f0e87e741f7928fff3c39ba77b99e7d1f2923c98c7bee052"} Nov 03 21:58:32.192942 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:32.192910 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" event={"ID":"6fa25de6-4fd2-4a96-9205-32d2d02b5642","Type":"ContainerStarted","Data":"06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417"} Nov 03 21:58:32.193302 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:32.192964 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:32.309751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:32.309665 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" podStartSLOduration=2.490512691 podStartE2EDuration="3.30965179s" totalImagesPullingTime="819.139099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:32.306101513 +0000 UTC m=+2579.916949327" watchObservedRunningTime="2026-11-03 21:58:32.30965179 +0000 UTC m=+2579.920499604" Nov 03 21:58:33.068028 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:33.067993 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:58:35.280251 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:35.280217 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:38.204017 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:38.203990 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 21:58:40.124081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:40.124049 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:44.348055 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:44.348019 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:45.351492 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.351460 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f"] Nov 03 21:58:45.351961 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.351666 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" podUID="aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" containerName="maas-api" containerID="cri-o://0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457" gracePeriod=30 Nov 03 21:58:45.553160 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.553090 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:45.656531 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.656506 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:58:45.763771 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.763740 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-openshift-service-ca\") pod \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " Nov 03 21:58:45.763946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.763824 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-metrics-certs\") pod \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " Nov 03 21:58:45.763946 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.763847 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls\") pod \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " Nov 03 21:58:45.764141 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.763977 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-kube-api-access-n68dz\" (UniqueName: \"kubernetes.io/projected/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-kube-api-access-n68dz\") pod \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\" (UID: \"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d\") " Nov 03 21:58:45.764141 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.764075 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-openshift-service-ca" pod "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" (UID: "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:58:45.764309 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.764292 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:45.766255 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.766232 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls" pod "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" (UID: "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:58:45.766503 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.766482 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-kube-api-access-n68dz" pod "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" (UID: "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d"). InnerVolumeSpecName "kube-api-access-n68dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:58:45.766587 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.766541 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-metrics-certs" pod "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" (UID: "aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:58:45.865483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.865455 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68dz\" (UniqueName: \"kubernetes.io/projected/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-kube-api-access-n68dz\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:45.865483 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.865481 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:45.865698 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.865491 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:58:45.887053 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:45.886959 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:46.201163 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.201058 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:46.302764 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.302732 5931 generic.go:356] "Generic (PLEG): container finished" podID="aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" containerID="0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457" exitCode=0 Nov 03 21:58:46.302962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.302781 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" Nov 03 21:58:46.302962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.302811 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" event={"ID":"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d","Type":"ContainerDied","Data":"0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457"} Nov 03 21:58:46.302962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.302867 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f" event={"ID":"aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d","Type":"ContainerDied","Data":"2726572d3d847187af966c75f9e6c6b509d2aeceab848c459c8f8145e22d825f"} Nov 03 21:58:46.302962 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.302887 5931 scope.go:122] "RemoveContainer" containerID="0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457" Nov 03 21:58:46.319604 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.319585 5931 scope.go:122] "RemoveContainer" containerID="0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457" Nov 03 21:58:46.319912 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:58:46.319886 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457\": container with ID starting with 0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457 not found: ID does not exist" containerID="0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457" Nov 03 21:58:46.320035 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.319922 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457"} err="failed to get container status \"0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457\": rpc error: code = NotFound desc = could not find container \"0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457\": container with ID starting with 0a15324ab4cdbe63db927761f0849f0ef418a29ab2fca9bba117029a19a69457 not found: ID does not exist" Nov 03 21:58:46.466185 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.466104 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f"] Nov 03 21:58:46.470084 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.470057 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c993b8-7d7b449c99-5ks2f"] Nov 03 21:58:46.505724 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.505697 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" path="/var/lib/kubelet/pods/aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d/volumes" Nov 03 21:58:46.989879 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:46.989851 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:47.209197 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:47.209162 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:47.451254 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:47.451219 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:47.665235 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:47.665197 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:48.240896 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:48.240865 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:50.584382 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:50.584351 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:52.349888 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:52.349861 5931 generic.go:356] "Generic (PLEG): container finished" podID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerID="e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5" exitCode=0 Nov 03 21:58:52.350315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:52.349919 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" event={"ID":"2a52b5bb-445f-410b-a621-9bebdb791b55","Type":"ContainerDied","Data":"e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5"} Nov 03 21:58:52.350315 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:52.350194 5931 scope.go:122] "RemoveContainer" containerID="e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5" Nov 03 21:58:52.351425 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:52.351396 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:58:52.351505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:52.351460 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:58:52.351505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:52.351493 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 21:58:52.950200 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:52.950145 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:53.207572 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:53.207490 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:53.359233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:53.359198 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" event={"ID":"2a52b5bb-445f-410b-a621-9bebdb791b55","Type":"ContainerStarted","Data":"11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d"} Nov 03 21:58:53.371469 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:53.371442 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:58:53.371623 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:53.371582 5931 patch_prober.go:29] interesting pod/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:58:53.371688 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:53.371641 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:58:53.465776 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:53.465652 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:54.299887 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:54.299856 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:54.371881 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:54.371847 5931 patch_prober.go:29] interesting pod/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:58:54.372340 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:54.371895 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:58:54.549658 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:54.544167 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:54.775044 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:54.775016 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:55.371577 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:55.371544 5931 patch_prober.go:29] interesting pod/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:58:55.371760 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:55.371596 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:58:56.372470 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:56.372435 5931 patch_prober.go:29] interesting pod/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:58:56.372982 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:56.372493 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:58:57.371967 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:57.371938 5931 patch_prober.go:29] interesting pod/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:58:57.372162 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:57.371988 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:58:58.187519 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:58.187492 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:58.376332 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:58.376302 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:58:58.376730 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:58.376709 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:58:58.378177 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:58:58.378153 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 21:59:03.067441 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:59:03.067400 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:59:03.270868 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:03.270836 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:07.954820 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:07.954781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:08.177547 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:08.177515 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:08.410720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:08.410693 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:12.973367 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:12.973153 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:13.206735 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:13.206700 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:13.435108 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:13.435079 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:18.025576 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:18.025542 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:23.007703 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:23.007662 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:25.975354 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:25.975323 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:27.291899 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:27.291866 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:28.013129 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:28.013100 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:28.285597 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:28.285516 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:28.653842 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:28.653808 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:31.893121 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:31.893093 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:32.126578 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:32.126548 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:33.068132 ip-10-0-128-179 kubenswrapper[5931]: E1103 21:59:33.068095 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 21:59:34.626433 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:34.626401 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:36.534666 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.532900 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:36.773505 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.773473 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:36.834717 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.834647 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml"] Nov 03 21:59:36.835180 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.835168 5931 cpu_manager.go:414] "containerMap: removing container" podUID="aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" containerName="maas-api" Nov 03 21:59:36.835233 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.835181 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" containerName="maas-api" Nov 03 21:59:36.835281 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.835272 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="aebc81d1-e4d2-4526-95d5-b4b10b0d3f7d" containerName="maas-api" Nov 03 21:59:36.838098 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.838082 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:36.873279 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.873256 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b252e860-5da3-4066-89c9-e575c6c66d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:36.873402 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.873382 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8vn\" (UniqueName: \"kubernetes.io/projected/b252e860-5da3-4066-89c9-e575c6c66d2f-kube-api-access-qp8vn\") pod \"payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:36.940488 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.940452 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7990bd\"" Nov 03 21:59:36.940708 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.940665 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7990bd-dockercfg-8v9b7\"" Nov 03 21:59:36.943422 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.943392 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml"] Nov 03 21:59:36.974547 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.974518 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b252e860-5da3-4066-89c9-e575c6c66d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:36.974714 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.974663 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8vn\" (UniqueName: \"kubernetes.io/projected/b252e860-5da3-4066-89c9-e575c6c66d2f-kube-api-access-qp8vn\") pod \"payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:36.975299 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.975281 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b252e860-5da3-4066-89c9-e575c6c66d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:36.982010 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.981984 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr"] Nov 03 21:59:36.984779 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:36.984761 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.031229 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.031207 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8vn\" (UniqueName: \"kubernetes.io/projected/b252e860-5da3-4066-89c9-e575c6c66d2f-kube-api-access-qp8vn\") pod \"payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:37.042189 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.042168 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr"] Nov 03 21:59:37.075985 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.075958 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjcdv\" (UniqueName: \"kubernetes.io/projected/ab4743a3-60ad-4f7c-82a8-f72018ef061f-kube-api-access-wjcdv\") pod \"payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.076081 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.076024 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4743a3-60ad-4f7c-82a8-f72018ef061f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.149472 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.149440 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:37.177636 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.177586 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjcdv\" (UniqueName: \"kubernetes.io/projected/ab4743a3-60ad-4f7c-82a8-f72018ef061f-kube-api-access-wjcdv\") pod \"payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.177746 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.177681 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4743a3-60ad-4f7c-82a8-f72018ef061f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.178312 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.178289 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4743a3-60ad-4f7c-82a8-f72018ef061f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.234048 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.234018 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjcdv\" (UniqueName: \"kubernetes.io/projected/ab4743a3-60ad-4f7c-82a8-f72018ef061f-kube-api-access-wjcdv\") pod \"payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.296738 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.296708 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:37.519872 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.519850 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml"] Nov 03 21:59:37.522165 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:59:37.522137 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb252e860_5da3_4066_89c9_e575c6c66d2f.slice/crio-644c3abf34135266c3dade1a5682ddbaf280ca1eeda7a44ef46baff74727237f WatchSource:0}: Error finding container 644c3abf34135266c3dade1a5682ddbaf280ca1eeda7a44ef46baff74727237f: Status 404 returned error can't find the container with id 644c3abf34135266c3dade1a5682ddbaf280ca1eeda7a44ef46baff74727237f Nov 03 21:59:37.626824 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.626788 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" event={"ID":"b252e860-5da3-4066-89c9-e575c6c66d2f","Type":"ContainerStarted","Data":"143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101"} Nov 03 21:59:37.627225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.626867 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:37.627225 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.626893 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" event={"ID":"b252e860-5da3-4066-89c9-e575c6c66d2f","Type":"ContainerStarted","Data":"644c3abf34135266c3dade1a5682ddbaf280ca1eeda7a44ef46baff74727237f"} Nov 03 21:59:37.670446 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.670424 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr"] Nov 03 21:59:37.672223 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:59:37.672200 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4743a3_60ad_4f7c_82a8_f72018ef061f.slice/crio-dbe05209f0ed8d83e7bb840b71783085e126b21bfce28dcd3c41cb91e4b4ea6d WatchSource:0}: Error finding container dbe05209f0ed8d83e7bb840b71783085e126b21bfce28dcd3c41cb91e4b4ea6d: Status 404 returned error can't find the container with id dbe05209f0ed8d83e7bb840b71783085e126b21bfce28dcd3c41cb91e4b4ea6d Nov 03 21:59:37.739431 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:37.739379 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" podStartSLOduration=1.739361486 podStartE2EDuration="1.739361486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:37.737757076 +0000 UTC m=+2645.348604923" watchObservedRunningTime="2026-11-03 21:59:37.739361486 +0000 UTC m=+2645.350209301" Nov 03 21:59:38.634781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:38.634744 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" event={"ID":"ab4743a3-60ad-4f7c-82a8-f72018ef061f","Type":"ContainerStarted","Data":"ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6"} Nov 03 21:59:38.635178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:38.634800 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" event={"ID":"ab4743a3-60ad-4f7c-82a8-f72018ef061f","Type":"ContainerStarted","Data":"dbe05209f0ed8d83e7bb840b71783085e126b21bfce28dcd3c41cb91e4b4ea6d"} Nov 03 21:59:38.635178 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:38.634843 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:38.747721 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:38.747673 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" podStartSLOduration=2.747660735 podStartE2EDuration="2.747660735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:38.745389451 +0000 UTC m=+2646.356237265" watchObservedRunningTime="2026-11-03 21:59:38.747660735 +0000 UTC m=+2646.358508549" Nov 03 21:59:39.898520 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:39.898455 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:41.707444 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:41.707417 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:44.905651 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:44.905587 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:47.271720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:47.271688 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:48.637751 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:48.637725 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 21:59:49.644087 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:49.644062 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 21:59:50.177196 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:50.177167 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:50.902759 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:50.902733 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:52.975568 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:52.975537 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5"] Nov 03 21:59:52.979720 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:52.979703 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.029545 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.029522 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-model-cache\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.029690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.029648 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-home\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.029690 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.029676 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tmp-dir\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.029796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.029711 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcflx\" (UniqueName: \"kubernetes.io/projected/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-kube-api-access-dcflx\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.029796 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.029783 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tls-certs\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.029883 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.029830 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-dshm\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.032781 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.032755 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-eb3520\"/\"openshift-service-ca.crt\"" Nov 03 21:59:53.032929 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.032833 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-eb3520\"/\"kube-root-ca.crt\"" Nov 03 21:59:53.082217 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.082194 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-eb3520\"/\"default-dockercfg-k2m5r\"" Nov 03 21:59:53.082465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.082452 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-eb3520\"/\"praxis-route-eb3520-kserve-self-signed-certs\"" Nov 03 21:59:53.085148 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.085128 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5"] Nov 03 21:59:53.130727 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.130699 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-model-cache\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.130865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.130796 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-home\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.130865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.130823 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tmp-dir\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.130865 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.130849 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcflx\" (UniqueName: \"kubernetes.io/projected/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-kube-api-access-dcflx\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.131027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.130897 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tls-certs\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.131027 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.130947 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-dshm\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.131142 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.131101 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-model-cache\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.131203 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.131158 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-home\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.131263 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.131218 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tmp-dir\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.133237 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.133210 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-dshm\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.133457 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.133439 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tls-certs\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.157571 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.157549 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:53.186662 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.186643 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcflx\" (UniqueName: \"kubernetes.io/projected/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-kube-api-access-dcflx\") pod \"praxis-route-eb3520-kserve-7b99659bc5-vj6b5\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.292810 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.292756 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:53.665465 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.665439 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5"] Nov 03 21:59:53.667599 ip-10-0-128-179 kubenswrapper[5931]: W1103 21:59:53.667571 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec04a1f5_a1b6_4604_a27f_633b30bf3e6a.slice/crio-686f2bfec2e6821e29434f0b0e4113d22018323ae37fa9cc236d655e62d81f60 WatchSource:0}: Error finding container 686f2bfec2e6821e29434f0b0e4113d22018323ae37fa9cc236d655e62d81f60: Status 404 returned error can't find the container with id 686f2bfec2e6821e29434f0b0e4113d22018323ae37fa9cc236d655e62d81f60 Nov 03 21:59:53.724811 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:53.724782 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" event={"ID":"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a","Type":"ContainerStarted","Data":"686f2bfec2e6821e29434f0b0e4113d22018323ae37fa9cc236d655e62d81f60"} Nov 03 21:59:54.731418 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:54.731389 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" event={"ID":"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a","Type":"ContainerStarted","Data":"2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11"} Nov 03 21:59:54.844584 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:54.844494 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" podStartSLOduration=2.844478543 podStartE2EDuration="2.844478543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:54.843471738 +0000 UTC m=+2662.454319552" watchObservedRunningTime="2026-11-03 21:59:54.844478543 +0000 UTC m=+2662.455326358" Nov 03 21:59:55.293058 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:55.293030 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 21:59:55.542804 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:55.542777 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:57.861339 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:57.861304 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:58.098110 ip-10-0-128-179 kubenswrapper[5931]: I1103 21:59:58.098084 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:00.215913 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:00.215877 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:03.068317 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:00:03.068277 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:00:03.293477 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:03.293447 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 22:00:03.305747 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:03.305721 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 22:00:03.798214 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:03.798184 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 22:00:04.228752 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.228718 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs"] Nov 03 22:00:04.229170 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.228916 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" podUID="a36c20a5-2d81-4faa-926c-65c516796e78" containerName="maas-api" containerID="cri-o://163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b" gracePeriod=30 Nov 03 22:00:04.433170 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.433078 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:04.482216 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.482158 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 22:00:04.639586 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.639553 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-metrics-certs\") pod \"a36c20a5-2d81-4faa-926c-65c516796e78\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " Nov 03 22:00:04.639785 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.639653 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls\") pod \"a36c20a5-2d81-4faa-926c-65c516796e78\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " Nov 03 22:00:04.639785 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.639773 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36c20a5-2d81-4faa-926c-65c516796e78-kube-api-access-z4npx\" (UniqueName: \"kubernetes.io/projected/a36c20a5-2d81-4faa-926c-65c516796e78-kube-api-access-z4npx\") pod \"a36c20a5-2d81-4faa-926c-65c516796e78\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " Nov 03 22:00:04.639918 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.639900 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36c20a5-2d81-4faa-926c-65c516796e78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36c20a5-2d81-4faa-926c-65c516796e78-openshift-service-ca\") pod \"a36c20a5-2d81-4faa-926c-65c516796e78\" (UID: \"a36c20a5-2d81-4faa-926c-65c516796e78\") " Nov 03 22:00:04.640508 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.640439 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36c20a5-2d81-4faa-926c-65c516796e78-openshift-service-ca" pod "a36c20a5-2d81-4faa-926c-65c516796e78" (UID: "a36c20a5-2d81-4faa-926c-65c516796e78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:04.642292 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.642269 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36c20a5-2d81-4faa-926c-65c516796e78-kube-api-access-z4npx" pod "a36c20a5-2d81-4faa-926c-65c516796e78" (UID: "a36c20a5-2d81-4faa-926c-65c516796e78"). InnerVolumeSpecName "kube-api-access-z4npx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:04.642571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.642553 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-metrics-certs" pod "a36c20a5-2d81-4faa-926c-65c516796e78" (UID: "a36c20a5-2d81-4faa-926c-65c516796e78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:04.642735 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.642714 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls" pod "a36c20a5-2d81-4faa-926c-65c516796e78" (UID: "a36c20a5-2d81-4faa-926c-65c516796e78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:04.741177 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.741105 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:04.741177 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.741133 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a36c20a5-2d81-4faa-926c-65c516796e78-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:04.741177 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.741146 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4npx\" (UniqueName: \"kubernetes.io/projected/a36c20a5-2d81-4faa-926c-65c516796e78-kube-api-access-z4npx\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:04.741177 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.741161 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36c20a5-2d81-4faa-926c-65c516796e78-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:04.794503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.794472 5931 generic.go:356] "Generic (PLEG): container finished" podID="a36c20a5-2d81-4faa-926c-65c516796e78" containerID="163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b" exitCode=0 Nov 03 22:00:04.794692 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.794522 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" Nov 03 22:00:04.794692 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.794559 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" event={"ID":"a36c20a5-2d81-4faa-926c-65c516796e78","Type":"ContainerDied","Data":"163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b"} Nov 03 22:00:04.794692 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.794632 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs" event={"ID":"a36c20a5-2d81-4faa-926c-65c516796e78","Type":"ContainerDied","Data":"3b73f20c38520fa9efaa7a6983ec4146ce3ea14c36d760df47fcc46a14cb439a"} Nov 03 22:00:04.794692 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.794662 5931 scope.go:122] "RemoveContainer" containerID="163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b" Nov 03 22:00:04.805833 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.805816 5931 scope.go:122] "RemoveContainer" containerID="163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b" Nov 03 22:00:04.806110 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:00:04.806094 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b\": container with ID starting with 163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b not found: ID does not exist" containerID="163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b" Nov 03 22:00:04.806155 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.806118 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b"} err="failed to get container status \"163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b\": rpc error: code = NotFound desc = could not find container \"163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b\": container with ID starting with 163106371f67c55a7ee598a66494a46734afc5e656e98aacbc80fa5e6922a89b not found: ID does not exist" Nov 03 22:00:04.956835 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.956812 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs"] Nov 03 22:00:04.960659 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:04.960635 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd2a9f-9867cfc99-bjpgs"] Nov 03 22:00:06.002243 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:06.002216 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:06.277062 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:06.276983 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:06.505959 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:06.505929 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36c20a5-2d81-4faa-926c-65c516796e78" path="/var/lib/kubelet/pods/a36c20a5-2d81-4faa-926c-65c516796e78/volumes" Nov 03 22:00:06.517453 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:06.517428 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:06.985194 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:06.985157 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:09.340144 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.340113 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd"] Nov 03 22:00:09.340651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.340635 5931 cpu_manager.go:414] "containerMap: removing container" podUID="a36c20a5-2d81-4faa-926c-65c516796e78" containerName="maas-api" Nov 03 22:00:09.340702 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.340653 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36c20a5-2d81-4faa-926c-65c516796e78" containerName="maas-api" Nov 03 22:00:09.340818 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.340809 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36c20a5-2d81-4faa-926c-65c516796e78" containerName="maas-api" Nov 03 22:00:09.345638 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.345603 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.447066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.447031 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:00:09.447066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.447037 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\"" Nov 03 22:00:09.451460 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.451436 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd"] Nov 03 22:00:09.493002 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.492978 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.493107 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.493042 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.493156 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.493131 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg842\" (UniqueName: \"kubernetes.io/projected/b4577cd3-9b4a-4712-8929-33664251dc2f-kube-api-access-kg842\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.594637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.594568 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.594756 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.594649 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.594756 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.594687 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg842\" (UniqueName: \"kubernetes.io/projected/b4577cd3-9b4a-4712-8929-33664251dc2f-kube-api-access-kg842\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.595337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.595318 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.597068 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.597043 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.650339 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.650317 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg842\" (UniqueName: \"kubernetes.io/projected/b4577cd3-9b4a-4712-8929-33664251dc2f-kube-api-access-kg842\") pod \"maas-api-revoke-sub-abeee6e5-dvdgd\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:09.658574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:09.658553 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:10.028072 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:10.028049 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd"] Nov 03 22:00:10.030870 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:00:10.030837 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4577cd3_9b4a_4712_8929_33664251dc2f.slice/crio-bdb9e7b705097605450d87f25492984690ef8fd84fe42b9865e8fbe10b5e292f WatchSource:0}: Error finding container bdb9e7b705097605450d87f25492984690ef8fd84fe42b9865e8fbe10b5e292f: Status 404 returned error can't find the container with id bdb9e7b705097605450d87f25492984690ef8fd84fe42b9865e8fbe10b5e292f Nov 03 22:00:10.838985 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:10.838949 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" event={"ID":"b4577cd3-9b4a-4712-8929-33664251dc2f","Type":"ContainerStarted","Data":"85da31fa2a269b76f514bb81377eb8a4fed83f0980bc2d727eb2a71fe50fea3b"} Nov 03 22:00:10.839343 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:10.839007 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" event={"ID":"b4577cd3-9b4a-4712-8929-33664251dc2f","Type":"ContainerStarted","Data":"bdb9e7b705097605450d87f25492984690ef8fd84fe42b9865e8fbe10b5e292f"} Nov 03 22:00:10.954322 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:10.954282 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" podStartSLOduration=1.954268043 podStartE2EDuration="1.954268043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:10.951969395 +0000 UTC m=+2678.562817208" watchObservedRunningTime="2026-11-03 22:00:10.954268043 +0000 UTC m=+2678.565115857" Nov 03 22:00:11.453004 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:11.452971 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:11.851463 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:11.851429 5931 generic.go:356] "Generic (PLEG): container finished" podID="b4577cd3-9b4a-4712-8929-33664251dc2f" containerID="85da31fa2a269b76f514bb81377eb8a4fed83f0980bc2d727eb2a71fe50fea3b" exitCode=0 Nov 03 22:00:11.851947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:11.851515 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" event={"ID":"b4577cd3-9b4a-4712-8929-33664251dc2f","Type":"ContainerDied","Data":"85da31fa2a269b76f514bb81377eb8a4fed83f0980bc2d727eb2a71fe50fea3b"} Nov 03 22:00:12.853931 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:12.853898 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:12.992692 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:12.992669 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:13.082709 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.082673 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:13.084670 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.084649 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx"] Nov 03 22:00:13.084841 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.084825 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" containerID="cri-o://99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1" gracePeriod=30 Nov 03 22:00:13.126502 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.126440 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-openshift-ca\") pod \"b4577cd3-9b4a-4712-8929-33664251dc2f\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " Nov 03 22:00:13.126502 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.126484 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-xks-ca\") pod \"b4577cd3-9b4a-4712-8929-33664251dc2f\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " Nov 03 22:00:13.126718 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.126654 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4577cd3-9b4a-4712-8929-33664251dc2f-kube-api-access-kg842\" (UniqueName: \"kubernetes.io/projected/b4577cd3-9b4a-4712-8929-33664251dc2f-kube-api-access-kg842\") pod \"b4577cd3-9b4a-4712-8929-33664251dc2f\" (UID: \"b4577cd3-9b4a-4712-8929-33664251dc2f\") " Nov 03 22:00:13.126863 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.126841 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-openshift-ca" pod "b4577cd3-9b4a-4712-8929-33664251dc2f" (UID: "b4577cd3-9b4a-4712-8929-33664251dc2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:13.127031 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.127010 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-openshift-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:13.128557 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.128531 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-xks-ca" pod "b4577cd3-9b4a-4712-8929-33664251dc2f" (UID: "b4577cd3-9b4a-4712-8929-33664251dc2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:13.128713 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.128694 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4577cd3-9b4a-4712-8929-33664251dc2f-kube-api-access-kg842" pod "b4577cd3-9b4a-4712-8929-33664251dc2f" (UID: "b4577cd3-9b4a-4712-8929-33664251dc2f"). InnerVolumeSpecName "kube-api-access-kg842". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:13.228393 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.228358 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4577cd3-9b4a-4712-8929-33664251dc2f-maas-api-xks-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:13.228393 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.228389 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg842\" (UniqueName: \"kubernetes.io/projected/b4577cd3-9b4a-4712-8929-33664251dc2f-kube-api-access-kg842\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:13.303711 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.303680 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:13.864756 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.864722 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" Nov 03 22:00:13.865155 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.864720 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd" event={"ID":"b4577cd3-9b4a-4712-8929-33664251dc2f","Type":"ContainerDied","Data":"bdb9e7b705097605450d87f25492984690ef8fd84fe42b9865e8fbe10b5e292f"} Nov 03 22:00:13.865155 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:13.864855 5931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb9e7b705097605450d87f25492984690ef8fd84fe42b9865e8fbe10b5e292f" Nov 03 22:00:16.267334 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:16.267303 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:18.335116 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.335094 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 22:00:18.374454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374276 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-certs\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374329 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-envoy\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374380 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istiod-ca-cert\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374443 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-kube-api-access-4c6jr\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-kube-api-access-4c6jr\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374495 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-socket\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374570 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-podinfo\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374653 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-credential-socket\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374655 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-certs" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:18.374844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374711 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-data\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.374844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.374769 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-token\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-token\") pod \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\" (UID: \"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791\") " Nov 03 22:00:18.376676 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.375176 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-socket" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:18.376676 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.375200 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.376676 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.375363 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-credential-socket" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:18.376676 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.375605 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-data" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:18.376676 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.376034 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istiod-ca-cert" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:18.378796 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.378771 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-envoy" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:18.378993 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.378976 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-kube-api-access-4c6jr" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "kube-api-access-4c6jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:18.379579 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.379552 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-token" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:18.379717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.379676 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-podinfo" pod "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" (UID: "dd9783f1-ed5d-440c-99c7-1c9b0f8a7791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:00:18.476814 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476786 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6jr\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-kube-api-access-4c6jr\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.476814 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476815 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.476987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476824 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.476987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476834 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.476987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476843 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.476987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476851 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.476987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476859 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.476987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.476870 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.899995 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.899962 5931 generic.go:356] "Generic (PLEG): container finished" podID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerID="99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1" exitCode=0 Nov 03 22:00:18.900152 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.900015 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" Nov 03 22:00:18.900152 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.900048 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" event={"ID":"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791","Type":"ContainerDied","Data":"99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1"} Nov 03 22:00:18.900152 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.900103 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx" event={"ID":"dd9783f1-ed5d-440c-99c7-1c9b0f8a7791","Type":"ContainerDied","Data":"02513e4daff6d1002212d4eac465f86343fba06dc1c2a95b8e0293ed03ca71bd"} Nov 03 22:00:18.900152 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.900132 5931 scope.go:122] "RemoveContainer" containerID="99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1" Nov 03 22:00:18.910098 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.910082 5931 scope.go:122] "RemoveContainer" containerID="6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab" Nov 03 22:00:18.918985 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.918968 5931 scope.go:122] "RemoveContainer" containerID="99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1" Nov 03 22:00:18.919244 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:00:18.919226 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1\": container with ID starting with 99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1 not found: ID does not exist" containerID="99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1" Nov 03 22:00:18.919306 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.919250 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1"} err="failed to get container status \"99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1\": rpc error: code = NotFound desc = could not find container \"99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1\": container with ID starting with 99c414cecb9e4476a8a4ace025dccf859f1b5c52cb9e32c8550c1c2daa027ec1 not found: ID does not exist" Nov 03 22:00:18.919306 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.919266 5931 scope.go:122] "RemoveContainer" containerID="6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab" Nov 03 22:00:18.919503 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:00:18.919486 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab\": container with ID starting with 6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab not found: ID does not exist" containerID="6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab" Nov 03 22:00:18.919543 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:18.919519 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab"} err="failed to get container status \"6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab\": rpc error: code = NotFound desc = could not find container \"6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab\": container with ID starting with 6dd5e7569eefa1c96196fe34c592bb85c0297d7106f4480e86f8c3cb6803dbab not found: ID does not exist" Nov 03 22:00:19.066377 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:19.066355 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx"] Nov 03 22:00:19.115917 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:19.115894 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd2a9f-openshift-default-5b99fdc989-tg7nx"] Nov 03 22:00:20.506801 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:20.506769 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" path="/var/lib/kubelet/pods/dd9783f1-ed5d-440c-99c7-1c9b0f8a7791/volumes" Nov 03 22:00:21.399713 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:21.399682 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:25.052216 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:25.052188 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:26.237923 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:26.237897 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:26.550840 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:26.550753 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:27.316196 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316162 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98"] Nov 03 22:00:27.316734 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316720 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b4577cd3-9b4a-4712-8929-33664251dc2f" containerName="revoke-keys" Nov 03 22:00:27.316800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316736 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4577cd3-9b4a-4712-8929-33664251dc2f" containerName="revoke-keys" Nov 03 22:00:27.316800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316766 5931 cpu_manager.go:414] "containerMap: removing container" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" Nov 03 22:00:27.316800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316771 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" Nov 03 22:00:27.316800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316788 5931 cpu_manager.go:414] "containerMap: removing container" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" Nov 03 22:00:27.316800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316793 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" Nov 03 22:00:27.316978 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316869 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4577cd3-9b4a-4712-8929-33664251dc2f" containerName="revoke-keys" Nov 03 22:00:27.316978 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.316880 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" Nov 03 22:00:27.321690 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.321674 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358441 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358409 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-data\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358539 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358469 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-certs\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358588 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358561 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-podinfo\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358666 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358652 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-socket\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358707 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358689 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-credential-socket\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358745 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358728 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-token\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358787 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358763 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b030ce95-ff95-417d-b111-c1fbdba2c301-istiod-ca-cert\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358825 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358811 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-envoy\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.358876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.358860 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pd7\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-kube-api-access-48pd7\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.373785 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.373759 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5bd35c-openshift-default-dockercfg-c6xxx\"" Nov 03 22:00:27.426327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.426301 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98"] Nov 03 22:00:27.459503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459479 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-socket\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459650 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459525 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-credential-socket\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459650 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459549 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-token\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459650 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459593 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b030ce95-ff95-417d-b111-c1fbdba2c301-istiod-ca-cert\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459819 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459655 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-envoy\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459819 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459719 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pd7\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-kube-api-access-48pd7\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459819 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459777 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-data\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459971 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459836 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-certs\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459971 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459864 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-podinfo\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459971 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459938 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-socket\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.459971 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.459957 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-credential-socket\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.460219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.460196 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-data\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.460279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.460232 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-certs\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.460389 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.460365 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b030ce95-ff95-417d-b111-c1fbdba2c301-istiod-ca-cert\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.462050 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.462027 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-envoy\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.462341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.462321 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-podinfo\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.516865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.516834 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-token\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.516994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.516935 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pd7\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-kube-api-access-48pd7\") pod \"e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.529653 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.529631 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:27.635037 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.635002 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:27.768304 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:27.768254 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:28.066339 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:28.066316 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98"] Nov 03 22:00:28.068312 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:00:28.068275 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb030ce95_ff95_417d_b111_c1fbdba2c301.slice/crio-fcc16b0aa9bb1e25a418ae01371012684f1ff703972442c57be5645952c8781f WatchSource:0}: Error finding container fcc16b0aa9bb1e25a418ae01371012684f1ff703972442c57be5645952c8781f: Status 404 returned error can't find the container with id fcc16b0aa9bb1e25a418ae01371012684f1ff703972442c57be5645952c8781f Nov 03 22:00:28.070485 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:28.070455 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:00:28.070594 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:28.070527 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:00:28.070594 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:28.070571 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:00:28.959866 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:28.959833 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" event={"ID":"b030ce95-ff95-417d-b111-c1fbdba2c301","Type":"ContainerStarted","Data":"362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259"} Nov 03 22:00:28.960320 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:28.959876 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" event={"ID":"b030ce95-ff95-417d-b111-c1fbdba2c301","Type":"ContainerStarted","Data":"fcc16b0aa9bb1e25a418ae01371012684f1ff703972442c57be5645952c8781f"} Nov 03 22:00:29.076104 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:29.076065 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podStartSLOduration=2.076049308 podStartE2EDuration="2.076049308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:29.074284841 +0000 UTC m=+2696.685132655" watchObservedRunningTime="2026-11-03 22:00:29.076049308 +0000 UTC m=+2696.686897124" Nov 03 22:00:29.635292 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:29.635265 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:29.640734 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:29.640710 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:29.967242 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:29.967178 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:29.967948 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:29.967932 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:00:30.374313 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.374286 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv"] Nov 03 22:00:30.374956 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.374941 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9783f1-ed5d-440c-99c7-1c9b0f8a7791" containerName="istio-proxy" Nov 03 22:00:30.378464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.378442 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.439665 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.439622 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv"] Nov 03 22:00:30.485389 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.485360 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6de8e6-openshift-default-dockercfg-vxss9\"" Nov 03 22:00:30.492867 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.492845 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.492997 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.492886 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.492997 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.492920 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.493111 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.493058 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.493111 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.493099 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfs6\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-kube-api-access-tzfs6\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.493228 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.493205 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.493269 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.493245 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.493353 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.493337 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.493390 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.493367 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594068 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594032 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594100 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594309 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594279 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594372 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594352 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594443 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594383 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfs6\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-kube-api-access-tzfs6\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594443 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594433 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594551 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594480 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594551 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594499 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594551 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594528 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594750 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594606 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.594750 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.594669 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.595111 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.595093 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.595337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.595312 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.595459 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.595368 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.596940 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.596913 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.597082 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.597035 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.651025 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.650975 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfs6\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-kube-api-access-tzfs6\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.651350 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.651325 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:30.691903 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:30.691879 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:31.154543 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:31.154519 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv"] Nov 03 22:00:31.156730 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:00:31.156702 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b6e331_b678_49ae_bd5f_9c2dba97ece7.slice/crio-2f903efa596b7c16633e5e0299b09427684ce548fc9feb7b321e4f4a2178ac28 WatchSource:0}: Error finding container 2f903efa596b7c16633e5e0299b09427684ce548fc9feb7b321e4f4a2178ac28: Status 404 returned error can't find the container with id 2f903efa596b7c16633e5e0299b09427684ce548fc9feb7b321e4f4a2178ac28 Nov 03 22:00:31.159151 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:31.159120 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:00:31.159232 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:31.159183 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:00:31.159232 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:31.159212 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:00:31.697030 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:31.696994 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:31.982310 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:31.982221 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" event={"ID":"68b6e331-b678-49ae-bd5f-9c2dba97ece7","Type":"ContainerStarted","Data":"65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e"} Nov 03 22:00:31.982310 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:31.982287 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" event={"ID":"68b6e331-b678-49ae-bd5f-9c2dba97ece7","Type":"ContainerStarted","Data":"2f903efa596b7c16633e5e0299b09427684ce548fc9feb7b321e4f4a2178ac28"} Nov 03 22:00:32.007276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:32.007243 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:32.111585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:32.111528 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podStartSLOduration=2.111511099 podStartE2EDuration="2.111511099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:32.110404386 +0000 UTC m=+2699.721252200" watchObservedRunningTime="2026-11-03 22:00:32.111511099 +0000 UTC m=+2699.722358916" Nov 03 22:00:32.255530 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:32.255450 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:32.692301 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:32.692264 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:32.693955 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:32.693931 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:00:32.694087 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:32.693972 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:00:33.072800 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:00:33.072723 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:00:33.692990 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:33.692960 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:00:33.693369 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:33.693017 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:00:34.692211 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:34.692177 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:00:34.692386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:34.692230 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:00:35.675307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:35.675277 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:00:35.683483 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:35.683459 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:00:35.692244 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:35.692220 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:00:35.692337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:35.692265 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:00:36.676565 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.676521 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z"] Nov 03 22:00:36.682408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.682385 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.692364 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.692338 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:00:36.692480 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.692380 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:00:36.751370 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.751338 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z"] Nov 03 22:00:36.781485 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.781455 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e549544-2c68-4acb-b081-8132b43b127e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.781668 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.781512 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.781668 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.781536 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblbr\" (UniqueName: \"kubernetes.io/projected/6e549544-2c68-4acb-b081-8132b43b127e-kube-api-access-gblbr\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.781796 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.781774 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-metrics-certs\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.793436 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.793412 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7990bd-serving-cert\"" Nov 03 22:00:36.883414 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.883378 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.883414 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.883420 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblbr\" (UniqueName: \"kubernetes.io/projected/6e549544-2c68-4acb-b081-8132b43b127e-kube-api-access-gblbr\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.883699 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:00:36.883520 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-serving-cert: secret "maas-api-e2e-praxis-b-7990bd-serving-cert" not found Nov 03 22:00:36.883699 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.883564 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-metrics-certs\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.883699 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:00:36.883577 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls podName:6e549544-2c68-4acb-b081-8132b43b127e nodeName:}" failed. No retries permitted until 2026-11-03 22:00:37.38356113 +0000 UTC m=+2704.994408924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls") pod "maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" (UID: "6e549544-2c68-4acb-b081-8132b43b127e") : secret "maas-api-e2e-praxis-b-7990bd-serving-cert" not found Nov 03 22:00:36.883699 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.883693 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e549544-2c68-4acb-b081-8132b43b127e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.884458 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.884434 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e549544-2c68-4acb-b081-8132b43b127e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.886011 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.885988 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-metrics-certs\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.942567 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.942492 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblbr\" (UniqueName: \"kubernetes.io/projected/6e549544-2c68-4acb-b081-8132b43b127e-kube-api-access-gblbr\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:36.962574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:36.962551 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:37.187908 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.187879 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:37.389231 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.389194 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:37.391600 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.391568 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:37.415805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.415780 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:37.595299 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.595270 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:37.692685 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.692600 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:00:37.692685 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.692677 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:00:37.748772 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.739246 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:37.975517 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.975494 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z"] Nov 03 22:00:37.978989 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:00:37.978953 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e549544_2c68_4acb_b081_8132b43b127e.slice/crio-e19cda067fe2ed2a188a7ae981cfe0e4127cbfa43db14ca25d1bd87cd81d740a WatchSource:0}: Error finding container e19cda067fe2ed2a188a7ae981cfe0e4127cbfa43db14ca25d1bd87cd81d740a: Status 404 returned error can't find the container with id e19cda067fe2ed2a188a7ae981cfe0e4127cbfa43db14ca25d1bd87cd81d740a Nov 03 22:00:37.979204 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.979176 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:37.980795 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:37.980772 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:00:38.024155 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:38.024131 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" event={"ID":"6e549544-2c68-4acb-b081-8132b43b127e","Type":"ContainerStarted","Data":"e19cda067fe2ed2a188a7ae981cfe0e4127cbfa43db14ca25d1bd87cd81d740a"} Nov 03 22:00:38.696898 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:38.696869 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:38.697254 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:38.697236 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:38.697952 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:38.697937 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:00:39.043316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:39.043219 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" event={"ID":"6e549544-2c68-4acb-b081-8132b43b127e","Type":"ContainerStarted","Data":"3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a"} Nov 03 22:00:39.043691 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:39.043541 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:39.156688 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:39.156629 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" podStartSLOduration=2.307871935 podStartE2EDuration="3.156593212s" totalImagesPullingTime="848.721277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:39.15608332 +0000 UTC m=+2706.766931134" watchObservedRunningTime="2026-11-03 22:00:39.156593212 +0000 UTC m=+2706.767441026" Nov 03 22:00:41.988106 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:41.988034 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:45.054471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:45.054436 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:00:46.745830 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:46.745797 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:51.300262 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:51.300223 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:51.569688 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:51.569554 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:56.244562 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:00:56.244529 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:01.127493 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:01.127459 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:03.068294 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:01:03.068259 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:01:05.842077 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:05.842043 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:11.319661 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:11.319600 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:16.274384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:16.274351 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:21.160183 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:21.160148 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:25.956978 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:25.956942 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:30.741558 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:30.741524 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:33.067785 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:01:33.067749 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:01:35.347858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:35.347823 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:35.572438 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:35.572409 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:39.485726 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.485690 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp"] Nov 03 22:01:39.490725 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.490697 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589331 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589304 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589451 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589370 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589451 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589436 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589531 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589488 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179b9c69-3c06-4eca-add5-47001fb043d4-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589531 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589520 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589596 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589560 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179b9c69-3c06-4eca-add5-47001fb043d4-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589596 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589578 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589735 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589715 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltq7\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-kube-api-access-pltq7\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.589795 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.589778 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.591701 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.591680 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7ee809-openshift-default-dockercfg-qsllw\"" Nov 03 22:01:39.691349 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691318 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691501 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691368 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691501 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691406 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179b9c69-3c06-4eca-add5-47001fb043d4-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691501 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691437 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691722 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691500 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179b9c69-3c06-4eca-add5-47001fb043d4-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691722 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691529 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691722 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691576 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltq7\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-kube-api-access-pltq7\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691722 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691606 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691937 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691736 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.691937 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691876 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.692051 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691933 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.692051 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.691996 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.692158 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.692135 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.692274 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.692254 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179b9c69-3c06-4eca-add5-47001fb043d4-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.693960 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.693940 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.694046 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.694030 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179b9c69-3c06-4eca-add5-47001fb043d4-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.713874 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.713849 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp"] Nov 03 22:01:39.747570 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.747506 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.747744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.747728 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltq7\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-kube-api-access-pltq7\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:39.804774 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:39.804752 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:40.232500 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.232477 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp"] Nov 03 22:01:40.235080 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:01:40.235051 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179b9c69_3c06_4eca_add5_47001fb043d4.slice/crio-5b9a1fa91c9c67f2217d390a15555da68ee23a1ba20de5533ce7ba7c43f66eff WatchSource:0}: Error finding container 5b9a1fa91c9c67f2217d390a15555da68ee23a1ba20de5533ce7ba7c43f66eff: Status 404 returned error can't find the container with id 5b9a1fa91c9c67f2217d390a15555da68ee23a1ba20de5533ce7ba7c43f66eff Nov 03 22:01:40.237548 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.237518 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:01:40.237622 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.237581 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:01:40.237665 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.237632 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:01:40.338315 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.338280 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:40.443096 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.443065 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" event={"ID":"179b9c69-3c06-4eca-add5-47001fb043d4","Type":"ContainerStarted","Data":"57bde4386dd45ef1ff495a817abf74f8ae2d3b6fcb6cc9c0f44b1661cf5ddf2e"} Nov 03 22:01:40.443260 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.443120 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" event={"ID":"179b9c69-3c06-4eca-add5-47001fb043d4","Type":"ContainerStarted","Data":"5b9a1fa91c9c67f2217d390a15555da68ee23a1ba20de5533ce7ba7c43f66eff"} Nov 03 22:01:40.561427 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.561333 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podStartSLOduration=1.56132137 podStartE2EDuration="1.56132137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:40.556793239 +0000 UTC m=+2768.167641050" watchObservedRunningTime="2026-11-03 22:01:40.56132137 +0000 UTC m=+2768.172169184" Nov 03 22:01:40.805529 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:40.805489 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:41.806815 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:41.806779 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:01:41.807204 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:41.806825 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:01:42.805882 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:42.805847 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:01:42.806051 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:42.805894 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:01:43.805561 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:43.805529 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:01:43.805966 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:43.805577 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:01:44.805215 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:44.805177 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:01:44.805415 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:44.805240 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:01:45.197889 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:45.197858 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:45.805464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:45.805434 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:01:45.805699 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:45.805490 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:01:45.958381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:45.958348 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx"] Nov 03 22:01:45.962913 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:45.962895 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090378 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090339 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm59l\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-kube-api-access-nm59l\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090529 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090492 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090584 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090642 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b501f81c-7184-4bef-a245-aa82d97673d8-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090668 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090705 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b501f81c-7184-4bef-a245-aa82d97673d8-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090750 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090786 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.090958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.090837 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.160834 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.160802 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx"] Nov 03 22:01:46.191751 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.191725 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.191906 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.191810 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.191906 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.191842 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b501f81c-7184-4bef-a245-aa82d97673d8-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.191906 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.191869 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192224 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192196 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192228 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b501f81c-7184-4bef-a245-aa82d97673d8-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192280 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192308 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192356 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192430 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm59l\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-kube-api-access-nm59l\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192601 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192577 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b501f81c-7184-4bef-a245-aa82d97673d8-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192827 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192778 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.192927 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.192847 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.193049 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.193025 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.194738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.194711 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.194851 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.194755 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b501f81c-7184-4bef-a245-aa82d97673d8-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.261513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.261480 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.261936 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.261705 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm59l\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-kube-api-access-nm59l\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.277475 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.277446 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:46.710107 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.710081 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx"] Nov 03 22:01:46.711943 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:01:46.711919 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb501f81c_7184_4bef_a245_aa82d97673d8.slice/crio-bee537d49d9d9747a285d34fe120eb86f631da7448f82d9f8079ab95e30a701e WatchSource:0}: Error finding container bee537d49d9d9747a285d34fe120eb86f631da7448f82d9f8079ab95e30a701e: Status 404 returned error can't find the container with id bee537d49d9d9747a285d34fe120eb86f631da7448f82d9f8079ab95e30a701e Nov 03 22:01:46.714647 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.714588 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:01:46.714738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.714697 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:01:46.714782 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.714744 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:01:46.805564 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.805542 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:01:46.805680 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:46.805579 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:01:47.499934 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:47.495129 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" event={"ID":"b501f81c-7184-4bef-a245-aa82d97673d8","Type":"ContainerStarted","Data":"cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f"} Nov 03 22:01:47.499934 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:47.495195 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" event={"ID":"b501f81c-7184-4bef-a245-aa82d97673d8","Type":"ContainerStarted","Data":"bee537d49d9d9747a285d34fe120eb86f631da7448f82d9f8079ab95e30a701e"} Nov 03 22:01:47.665624 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:47.665557 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podStartSLOduration=2.665540418 podStartE2EDuration="2.665540418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:47.661340073 +0000 UTC m=+2775.272187888" watchObservedRunningTime="2026-11-03 22:01:47.665540418 +0000 UTC m=+2775.276388230" Nov 03 22:01:47.809380 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:47.809294 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:47.809751 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:47.809727 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:47.810515 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:47.810496 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:01:48.277931 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:48.277891 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:48.279363 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:48.279334 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:01:48.279483 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:48.279394 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 22:01:49.277891 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:49.277850 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:01:49.278353 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:49.277898 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 22:01:50.278622 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:50.278586 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:01:50.278999 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:50.278661 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 22:01:50.365803 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:50.365772 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:51.278811 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:51.278777 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:01:51.279283 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:51.278836 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 22:01:51.670435 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:51.670402 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:52.278458 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:52.278425 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:01:52.278664 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:52.278483 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 22:01:53.278392 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:53.278320 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 22:01:53.278773 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:53.278386 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 22:01:54.826598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:54.826559 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:54.827042 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:54.826982 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:54.827741 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:54.827713 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:01:55.142367 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:55.142326 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:55.459161 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:01:55.459052 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:00.479800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:00.479766 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:03.067774 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:02:03.067741 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:02:05.750278 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:05.750215 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:07.819163 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:07.819132 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t"] Nov 03 22:02:07.924145 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:07.924118 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc"] Nov 03 22:02:08.274291 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:08.274248 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:08.519950 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:08.519913 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:10.646947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:10.646904 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:12.820316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:12.820240 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" podUID="6ab0f586-7675-46c5-bebb-40b2a291a049" containerName="payload-processing" containerID="cri-o://95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196" gracePeriod=25 Nov 03 22:02:12.925324 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:12.925296 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" podUID="f06fe570-6375-402e-9484-119bfea6351e" containerName="payload-pre-processing" containerID="cri-o://253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249" gracePeriod=25 Nov 03 22:02:13.090522 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.090498 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 22:02:13.182708 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.182678 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab0f586-7675-46c5-bebb-40b2a291a049-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0f586-7675-46c5-bebb-40b2a291a049-plugins-config-volume\") pod \"6ab0f586-7675-46c5-bebb-40b2a291a049\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " Nov 03 22:02:13.182826 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.182729 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab0f586-7675-46c5-bebb-40b2a291a049-kube-api-access-4z8dm\" (UniqueName: \"kubernetes.io/projected/6ab0f586-7675-46c5-bebb-40b2a291a049-kube-api-access-4z8dm\") pod \"6ab0f586-7675-46c5-bebb-40b2a291a049\" (UID: \"6ab0f586-7675-46c5-bebb-40b2a291a049\") " Nov 03 22:02:13.183042 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.183022 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab0f586-7675-46c5-bebb-40b2a291a049-plugins-config-volume" pod "6ab0f586-7675-46c5-bebb-40b2a291a049" (UID: "6ab0f586-7675-46c5-bebb-40b2a291a049"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:13.183285 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.183267 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0f586-7675-46c5-bebb-40b2a291a049-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:13.184760 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.184740 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab0f586-7675-46c5-bebb-40b2a291a049-kube-api-access-4z8dm" pod "6ab0f586-7675-46c5-bebb-40b2a291a049" (UID: "6ab0f586-7675-46c5-bebb-40b2a291a049"). InnerVolumeSpecName "kube-api-access-4z8dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:13.192308 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.192288 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 22:02:13.283731 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.283704 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06fe570-6375-402e-9484-119bfea6351e-kube-api-access-f7bmn\" (UniqueName: \"kubernetes.io/projected/f06fe570-6375-402e-9484-119bfea6351e-kube-api-access-f7bmn\") pod \"f06fe570-6375-402e-9484-119bfea6351e\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " Nov 03 22:02:13.283958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.283929 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06fe570-6375-402e-9484-119bfea6351e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06fe570-6375-402e-9484-119bfea6351e-plugins-config-volume\") pod \"f06fe570-6375-402e-9484-119bfea6351e\" (UID: \"f06fe570-6375-402e-9484-119bfea6351e\") " Nov 03 22:02:13.284244 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.284224 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06fe570-6375-402e-9484-119bfea6351e-plugins-config-volume" pod "f06fe570-6375-402e-9484-119bfea6351e" (UID: "f06fe570-6375-402e-9484-119bfea6351e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:13.284306 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.284241 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8dm\" (UniqueName: \"kubernetes.io/projected/6ab0f586-7675-46c5-bebb-40b2a291a049-kube-api-access-4z8dm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:13.285808 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.285784 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06fe570-6375-402e-9484-119bfea6351e-kube-api-access-f7bmn" pod "f06fe570-6375-402e-9484-119bfea6351e" (UID: "f06fe570-6375-402e-9484-119bfea6351e"). InnerVolumeSpecName "kube-api-access-f7bmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:13.385041 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.384981 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06fe570-6375-402e-9484-119bfea6351e-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:13.385041 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.385002 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bmn\" (UniqueName: \"kubernetes.io/projected/f06fe570-6375-402e-9484-119bfea6351e-kube-api-access-f7bmn\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:13.671607 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.671538 5931 generic.go:356] "Generic (PLEG): container finished" podID="f06fe570-6375-402e-9484-119bfea6351e" containerID="253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249" exitCode=0 Nov 03 22:02:13.671607 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.671575 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" Nov 03 22:02:13.671824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.671626 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" event={"ID":"f06fe570-6375-402e-9484-119bfea6351e","Type":"ContainerDied","Data":"253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249"} Nov 03 22:02:13.671824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.671678 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc" event={"ID":"f06fe570-6375-402e-9484-119bfea6351e","Type":"ContainerDied","Data":"fd230a5d8dbeb967c0919f0f67e9e09a54d8fbe092301502de3e8cac2de290b3"} Nov 03 22:02:13.671824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.671709 5931 scope.go:122] "RemoveContainer" containerID="253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249" Nov 03 22:02:13.673070 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.673046 5931 generic.go:356] "Generic (PLEG): container finished" podID="6ab0f586-7675-46c5-bebb-40b2a291a049" containerID="95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196" exitCode=0 Nov 03 22:02:13.673176 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.673095 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" event={"ID":"6ab0f586-7675-46c5-bebb-40b2a291a049","Type":"ContainerDied","Data":"95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196"} Nov 03 22:02:13.673176 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.673129 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" event={"ID":"6ab0f586-7675-46c5-bebb-40b2a291a049","Type":"ContainerDied","Data":"2169becaf37dd708fc627fb8d4896e197a400cce5ef285cc30146728ce6db6da"} Nov 03 22:02:13.673176 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.673096 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t" Nov 03 22:02:13.683130 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.683112 5931 scope.go:122] "RemoveContainer" containerID="253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249" Nov 03 22:02:13.683432 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:02:13.683402 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249\": container with ID starting with 253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249 not found: ID does not exist" containerID="253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249" Nov 03 22:02:13.683531 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.683436 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249"} err="failed to get container status \"253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249\": rpc error: code = NotFound desc = could not find container \"253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249\": container with ID starting with 253854ac91afbe333e653484b3574edd3fb7840fd04e699cf2b0daa91cb7c249 not found: ID does not exist" Nov 03 22:02:13.683531 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.683452 5931 scope.go:122] "RemoveContainer" containerID="95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196" Nov 03 22:02:13.693203 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.693185 5931 scope.go:122] "RemoveContainer" containerID="95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196" Nov 03 22:02:13.693440 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:02:13.693423 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196\": container with ID starting with 95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196 not found: ID does not exist" containerID="95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196" Nov 03 22:02:13.693478 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.693447 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196"} err="failed to get container status \"95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196\": rpc error: code = NotFound desc = could not find container \"95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196\": container with ID starting with 95800227a9dff980179a2802f9e02f6f40ccbe5a787bec12b7b7b8f4eeb05196 not found: ID does not exist" Nov 03 22:02:13.837877 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.837855 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t"] Nov 03 22:02:13.841924 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.841902 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c833-56d499fbdb-ds84t"] Nov 03 22:02:13.949204 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.949134 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc"] Nov 03 22:02:13.951651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:13.951629 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c833-7b64dc4f59-kkdlc"] Nov 03 22:02:14.506909 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:14.506875 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab0f586-7675-46c5-bebb-40b2a291a049" path="/var/lib/kubelet/pods/6ab0f586-7675-46c5-bebb-40b2a291a049/volumes" Nov 03 22:02:14.507388 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:14.507369 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06fe570-6375-402e-9484-119bfea6351e" path="/var/lib/kubelet/pods/f06fe570-6375-402e-9484-119bfea6351e/volumes" Nov 03 22:02:15.435204 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:15.435165 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:20.239741 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:20.239698 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:25.162354 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:25.162320 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:29.811563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:29.811523 5931 generic.go:356] "Generic (PLEG): container finished" podID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerID="362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259" exitCode=0 Nov 03 22:02:29.811998 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:29.811597 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" event={"ID":"b030ce95-ff95-417d-b111-c1fbdba2c301","Type":"ContainerDied","Data":"362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259"} Nov 03 22:02:29.811998 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:29.811942 5931 scope.go:122] "RemoveContainer" containerID="362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259" Nov 03 22:02:29.813284 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:29.813257 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:29.813375 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:29.813320 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:29.813375 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:29.813356 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:29.967290 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:29.967263 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:02:30.303934 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:30.303905 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:30.819357 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:30.819321 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" event={"ID":"b030ce95-ff95-417d-b111-c1fbdba2c301","Type":"ContainerStarted","Data":"19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97"} Nov 03 22:02:31.641926 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:31.641895 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:02:31.647065 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:31.647022 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:02:31.647203 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:31.647084 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:02:32.647927 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:32.647894 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:02:32.648412 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:32.647937 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:02:33.075360 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:02:33.075273 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:02:33.641636 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:33.640914 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:02:33.641636 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:33.640984 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:02:34.638072 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:34.638038 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:02:34.646767 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:34.646722 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:02:34.654836 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:34.654802 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:35.636025 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:35.635981 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:02:35.636240 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:35.636044 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:02:36.425711 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:36.425669 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:36.639765 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:36.639733 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:02:36.640189 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:36.640172 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:02:36.640849 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:36.640835 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:02:36.658735 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:36.658709 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:36.916739 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:36.916712 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:37.141860 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:37.141797 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:37.473869 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:37.473842 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:39.969662 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.969628 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw"] Nov 03 22:02:39.970196 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.970183 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6ab0f586-7675-46c5-bebb-40b2a291a049" containerName="payload-processing" Nov 03 22:02:39.970243 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.970197 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab0f586-7675-46c5-bebb-40b2a291a049" containerName="payload-processing" Nov 03 22:02:39.970243 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.970225 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f06fe570-6375-402e-9484-119bfea6351e" containerName="payload-pre-processing" Nov 03 22:02:39.970243 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.970230 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06fe570-6375-402e-9484-119bfea6351e" containerName="payload-pre-processing" Nov 03 22:02:39.970336 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.970320 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab0f586-7675-46c5-bebb-40b2a291a049" containerName="payload-processing" Nov 03 22:02:39.970336 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.970333 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06fe570-6375-402e-9484-119bfea6351e" containerName="payload-pre-processing" Nov 03 22:02:39.973460 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:39.973438 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056369 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056560 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056414 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056560 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056534 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056709 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056567 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056709 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056592 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4ab36-e83f-4782-a119-b50938bcf669-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056815 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056720 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqkn7\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-kube-api-access-bqkn7\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056815 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056779 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056917 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056889 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.056969 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.056938 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158439 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158677 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158490 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158677 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158533 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158677 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158563 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158677 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158596 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158880 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158686 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158880 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158818 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.158880 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158857 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4ab36-e83f-4782-a119-b50938bcf669-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.159042 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.158934 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqkn7\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-kube-api-access-bqkn7\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.159042 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.159018 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.159147 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.159049 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.159147 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.159103 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.159147 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.159129 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.159920 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.159896 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4ab36-e83f-4782-a119-b50938bcf669-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.161071 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.161046 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.161162 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.161141 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.211103 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.211060 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw"] Nov 03 22:02:40.215925 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.215898 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqkn7\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-kube-api-access-bqkn7\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.216077 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.215901 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.286407 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.286338 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:40.719098 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.719068 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw"] Nov 03 22:02:40.721486 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:02:40.721461 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb4ab36_e83f_4782_a119_b50938bcf669.slice/crio-a9646d4d0967c5cfffe5a047b45e20810815de719ed099d5f7ca4b58ab998319 WatchSource:0}: Error finding container a9646d4d0967c5cfffe5a047b45e20810815de719ed099d5f7ca4b58ab998319: Status 404 returned error can't find the container with id a9646d4d0967c5cfffe5a047b45e20810815de719ed099d5f7ca4b58ab998319 Nov 03 22:02:40.724488 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.724448 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:40.724580 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.724539 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:40.724643 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.724587 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:40.836722 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.836687 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:40.907046 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.907009 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" event={"ID":"9cb4ab36-e83f-4782-a119-b50938bcf669","Type":"ContainerStarted","Data":"5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1"} Nov 03 22:02:40.907229 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:40.907077 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" event={"ID":"9cb4ab36-e83f-4782-a119-b50938bcf669","Type":"ContainerStarted","Data":"a9646d4d0967c5cfffe5a047b45e20810815de719ed099d5f7ca4b58ab998319"} Nov 03 22:02:41.030554 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:41.030446 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podStartSLOduration=2.030427366 podStartE2EDuration="2.030427366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:41.029833351 +0000 UTC m=+2828.640681165" watchObservedRunningTime="2026-11-03 22:02:41.030427366 +0000 UTC m=+2828.641275180" Nov 03 22:02:41.287690 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:41.286937 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:41.304761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:41.303503 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 22:02:41.304761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:41.303564 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 22:02:41.449637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:41.446634 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:41.846636 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:41.844212 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:42.079174 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:42.079141 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:42.287432 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:42.287337 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 22:02:42.287432 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:42.287390 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 22:02:43.287045 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:43.287011 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 22:02:43.287442 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:43.287069 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 22:02:43.304896 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:43.304870 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:43.554255 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:43.554152 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:43.556834 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:43.556807 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792"] Nov 03 22:02:43.557067 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:43.557040 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" containerID="cri-o://11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d" gracePeriod=30 Nov 03 22:02:43.794777 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:43.794747 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:44.287019 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:44.286984 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 22:02:44.287221 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:44.287046 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 22:02:45.287732 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:45.287701 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 22:02:45.288126 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:45.287754 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 22:02:46.095330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.095295 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:46.287081 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.286960 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 22:02:46.287081 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.287053 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 22:02:46.352273 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.352138 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:46.451949 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.451909 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg"] Nov 03 22:02:46.456473 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.456443 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.570413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.570373 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.570771 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.570747 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1419cffe-7409-4586-85b7-a40ab774dffd-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.571005 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.570990 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.571141 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.571129 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1419cffe-7409-4586-85b7-a40ab774dffd-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.571284 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.571270 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.571441 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.571429 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.571591 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.571579 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.571760 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.571746 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtm25\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-kube-api-access-qtm25\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.571889 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.571876 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.577344 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.577296 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg"] Nov 03 22:02:46.673669 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673583 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.673669 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673639 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1419cffe-7409-4586-85b7-a40ab774dffd-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.673669 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673666 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.673979 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673722 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.673979 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673755 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.673979 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673794 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtm25\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-kube-api-access-qtm25\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.673979 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673826 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.673979 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.673955 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.674276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.674000 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1419cffe-7409-4586-85b7-a40ab774dffd-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.674276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.674111 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.674276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.674138 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.674455 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.674381 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.674727 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.674700 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1419cffe-7409-4586-85b7-a40ab774dffd-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.674727 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.674721 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.676107 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.676074 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.676556 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.676539 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1419cffe-7409-4586-85b7-a40ab774dffd-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.730541 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.730497 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.730541 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.730536 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtm25\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-kube-api-access-qtm25\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:46.772861 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:46.772830 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:47.204974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.204944 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg"] Nov 03 22:02:47.206939 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:02:47.206910 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1419cffe_7409_4586_85b7_a40ab774dffd.slice/crio-88e352c1256ec893fe2541621d3e66a8f9de293e1639ceab3176df70f5e7d884 WatchSource:0}: Error finding container 88e352c1256ec893fe2541621d3e66a8f9de293e1639ceab3176df70f5e7d884: Status 404 returned error can't find the container with id 88e352c1256ec893fe2541621d3e66a8f9de293e1639ceab3176df70f5e7d884 Nov 03 22:02:47.209494 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.209465 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:47.209575 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.209528 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:47.209575 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.209556 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:47.287604 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.287564 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 22:02:47.287811 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.287685 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 22:02:47.327159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.327120 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:47.976630 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.976562 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" event={"ID":"1419cffe-7409-4586-85b7-a40ab774dffd","Type":"ContainerStarted","Data":"8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46"} Nov 03 22:02:47.977232 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:47.977207 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" event={"ID":"1419cffe-7409-4586-85b7-a40ab774dffd","Type":"ContainerStarted","Data":"88e352c1256ec893fe2541621d3e66a8f9de293e1639ceab3176df70f5e7d884"} Nov 03 22:02:48.100018 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.099970 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podStartSLOduration=2.099955502 podStartE2EDuration="2.099955502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:48.096171172 +0000 UTC m=+2835.707018999" watchObservedRunningTime="2026-11-03 22:02:48.099955502 +0000 UTC m=+2835.710803316" Nov 03 22:02:48.291746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.291655 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:48.292071 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.292046 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:48.292712 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.292692 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:02:48.773122 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.773089 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:48.774648 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.774586 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:02:48.774753 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.774658 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:02:48.854803 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.854770 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 22:02:48.984778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.984746 5931 generic.go:356] "Generic (PLEG): container finished" podID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerID="11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d" exitCode=0 Nov 03 22:02:48.985250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.984801 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" event={"ID":"2a52b5bb-445f-410b-a621-9bebdb791b55","Type":"ContainerDied","Data":"11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d"} Nov 03 22:02:48.985250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.984829 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" Nov 03 22:02:48.985250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.984915 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792" event={"ID":"2a52b5bb-445f-410b-a621-9bebdb791b55","Type":"ContainerDied","Data":"4f5d4fdc04cf2ce02acf19ecb2e849f482e81c325fef3012f332fdbae73ae606"} Nov 03 22:02:48.985250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.984967 5931 scope.go:122] "RemoveContainer" containerID="11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d" Nov 03 22:02:48.995428 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:48.995406 5931 scope.go:122] "RemoveContainer" containerID="e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5" Nov 03 22:02:49.005365 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.005312 5931 scope.go:122] "RemoveContainer" containerID="11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d" Nov 03 22:02:49.005881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.005320 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a52b5bb-445f-410b-a621-9bebdb791b55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52b5bb-445f-410b-a621-9bebdb791b55-istiod-ca-cert\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.005881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.005654 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a52b5bb-445f-410b-a621-9bebdb791b55-istiod-ca-cert" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:49.005881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.005655 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-data\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.006844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.005909 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-data" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:49.006844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.006319 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-kube-api-access-24s55\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-kube-api-access-24s55\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.006844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.006408 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-socket\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.006844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.006525 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-podinfo\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.006844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.006574 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-token\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-token\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.006844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.006809 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-credential-socket\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.007143 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.007071 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-socket" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:49.007194 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.007153 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-certs\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.007487 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.007311 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-envoy\") pod \"2a52b5bb-445f-410b-a621-9bebdb791b55\" (UID: \"2a52b5bb-445f-410b-a621-9bebdb791b55\") " Nov 03 22:02:49.007700 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.007676 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-credential-socket" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:49.008115 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008097 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-certs" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:49.008304 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:02:49.008277 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d\": container with ID starting with 11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d not found: ID does not exist" containerID="11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d" Nov 03 22:02:49.008377 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008316 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d"} err="failed to get container status \"11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d\": rpc error: code = NotFound desc = could not find container \"11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d\": container with ID starting with 11d288de0571d562c9b58ed88680f086940122bb70ec29931c5f5d7da9ca751d not found: ID does not exist" Nov 03 22:02:49.008377 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008340 5931 scope.go:122] "RemoveContainer" containerID="e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5" Nov 03 22:02:49.008377 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008342 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.008634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008379 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.008634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008396 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.008634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008410 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52b5bb-445f-410b-a621-9bebdb791b55-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.008634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.008423 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.009391 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:02:49.009312 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5\": container with ID starting with e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5 not found: ID does not exist" containerID="e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5" Nov 03 22:02:49.009391 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.009346 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5"} err="failed to get container status \"e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5\": rpc error: code = NotFound desc = could not find container \"e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5\": container with ID starting with e6676597b45409f4e63dfe6d714f1c3e1f7be45acbc2dd4f89007646699fffa5 not found: ID does not exist" Nov 03 22:02:49.011205 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.011173 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-kube-api-access-24s55" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "kube-api-access-24s55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:49.011343 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.011320 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-token" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:49.013016 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.012930 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-envoy" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:49.013016 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.012991 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-podinfo" pod "2a52b5bb-445f-410b-a621-9bebdb791b55" (UID: "2a52b5bb-445f-410b-a621-9bebdb791b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:02:49.110341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.110258 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24s55\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-kube-api-access-24s55\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.110341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.110296 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.110341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.110310 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.110341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.110323 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52b5bb-445f-410b-a621-9bebdb791b55-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:02:49.408950 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.408869 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792"] Nov 03 22:02:49.413355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.413331 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c833-openshift-default-7dc6778cb9-7z792"] Nov 03 22:02:49.773587 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.773501 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:02:49.773587 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:49.773558 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:02:50.071976 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:50.071894 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:50.506193 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:50.506158 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" path="/var/lib/kubelet/pods/2a52b5bb-445f-410b-a621-9bebdb791b55/volumes" Nov 03 22:02:50.773606 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:50.773524 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:02:50.773606 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:50.773582 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:02:51.773441 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:51.773405 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:02:51.773901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:51.773463 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:02:52.119772 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:52.119737 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:52.773328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:52.773291 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:02:52.773528 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:52.773357 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:02:52.789065 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:52.789031 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:53.773850 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:53.773812 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:02:53.774244 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:53.773871 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:02:54.778383 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:54.778350 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:54.778794 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:54.778654 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:54.779395 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:54.779376 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:02:57.917890 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:57.917858 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:58.212419 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.212347 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:58.451950 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.451910 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:58.568447 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.568354 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z"] Nov 03 22:02:58.569043 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.569024 5931 cpu_manager.go:414] "containerMap: removing container" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" Nov 03 22:02:58.569102 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.569044 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" Nov 03 22:02:58.569102 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.569054 5931 cpu_manager.go:414] "containerMap: removing container" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" Nov 03 22:02:58.569102 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.569060 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" Nov 03 22:02:58.569209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.569167 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" Nov 03 22:02:58.569209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.569176 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a52b5bb-445f-410b-a621-9bebdb791b55" containerName="istio-proxy" Nov 03 22:02:58.572035 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.572019 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.630638 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.630593 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z"] Nov 03 22:02:58.644881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.644842 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15288224-fbef-47db-b0a5-8d4f488a6915-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.645067 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.645045 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5psk\" (UniqueName: \"kubernetes.io/projected/15288224-fbef-47db-b0a5-8d4f488a6915-kube-api-access-m5psk\") pod \"payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.675857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.675828 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-29dbd9\"" Nov 03 22:02:58.681709 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.681678 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw"] Nov 03 22:02:58.684565 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.684543 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:58.737971 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.737937 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-29dbd9-dockercfg-bfgpn\"" Nov 03 22:02:58.749224 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.749180 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15288224-fbef-47db-b0a5-8d4f488a6915-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.749461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.749356 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6q66\" (UniqueName: \"kubernetes.io/projected/85227cfc-1db8-46e0-964a-c24442389ad2-kube-api-access-d6q66\") pod \"payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:58.749461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.749403 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5psk\" (UniqueName: \"kubernetes.io/projected/15288224-fbef-47db-b0a5-8d4f488a6915-kube-api-access-m5psk\") pod \"payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.750111 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.749458 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85227cfc-1db8-46e0-964a-c24442389ad2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:58.751083 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.751053 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15288224-fbef-47db-b0a5-8d4f488a6915-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.770261 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.759126 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw"] Nov 03 22:02:58.806411 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.806381 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5psk\" (UniqueName: \"kubernetes.io/projected/15288224-fbef-47db-b0a5-8d4f488a6915-kube-api-access-m5psk\") pod \"payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.850439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.850401 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6q66\" (UniqueName: \"kubernetes.io/projected/85227cfc-1db8-46e0-964a-c24442389ad2-kube-api-access-d6q66\") pod \"payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:58.850604 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.850467 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85227cfc-1db8-46e0-964a-c24442389ad2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:58.851149 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.851131 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85227cfc-1db8-46e0-964a-c24442389ad2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:58.884251 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.884224 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:02:58.908084 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.908058 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6q66\" (UniqueName: \"kubernetes.io/projected/85227cfc-1db8-46e0-964a-c24442389ad2-kube-api-access-d6q66\") pod \"payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:58.997660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:58.997626 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:02:59.077807 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.077775 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x"] Nov 03 22:02:59.081828 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.081811 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.135958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.135864 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-a49a32-openshift-default-dockercfg-cnh26\"" Nov 03 22:02:59.140935 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.140903 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x"] Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.162712 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-podinfo\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.162843 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06145d74-f6d7-4895-8e8e-ad0582b1f784-istiod-ca-cert\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.162916 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-envoy\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.162946 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-credential-socket\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.163000 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-token\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.163045 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-socket\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.163071 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-certs\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.163116 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqdp\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-kube-api-access-bkqdp\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.163947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.163145 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-data\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265288 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.264412 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-envoy\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265288 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.264462 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-credential-socket\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265288 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.264516 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-token\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265288 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.265105 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-credential-socket\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265793 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.265712 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-socket\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265793 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.265774 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-certs\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.265830 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqdp\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-kube-api-access-bkqdp\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.265968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.265860 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-data\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.266067 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.266018 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-podinfo\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.266124 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.266075 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06145d74-f6d7-4895-8e8e-ad0582b1f784-istiod-ca-cert\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.266393 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.266367 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-socket\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.266480 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.266416 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-certs\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.266758 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.266733 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-data\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.267249 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.267228 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06145d74-f6d7-4895-8e8e-ad0582b1f784-istiod-ca-cert\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.269546 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.269507 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-podinfo\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.269792 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.269767 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-envoy\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.269903 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.269809 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z"] Nov 03 22:02:59.275455 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:02:59.275430 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15288224_fbef_47db_b0a5_8d4f488a6915.slice/crio-ec97caa5a80acad381cd97f076f1ad8bddaa49260b7c64f06cf0c734b78e333a WatchSource:0}: Error finding container ec97caa5a80acad381cd97f076f1ad8bddaa49260b7c64f06cf0c734b78e333a: Status 404 returned error can't find the container with id ec97caa5a80acad381cd97f076f1ad8bddaa49260b7c64f06cf0c734b78e333a Nov 03 22:02:59.321731 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.321705 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-token\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.321988 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.321968 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqdp\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-kube-api-access-bkqdp\") pod \"e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.338022 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.337977 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:59.380688 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.380658 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw"] Nov 03 22:02:59.383259 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:02:59.383231 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85227cfc_1db8_46e0_964a_c24442389ad2.slice/crio-3b65b5ead7b9116bcc3b8004af28eb3000824f24bf974a98ab2cbb176b112bfc WatchSource:0}: Error finding container 3b65b5ead7b9116bcc3b8004af28eb3000824f24bf974a98ab2cbb176b112bfc: Status 404 returned error can't find the container with id 3b65b5ead7b9116bcc3b8004af28eb3000824f24bf974a98ab2cbb176b112bfc Nov 03 22:02:59.400083 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.400047 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:02:59.559840 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.559808 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:59.836972 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.836941 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x"] Nov 03 22:02:59.837816 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:02:59.837789 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06145d74_f6d7_4895_8e8e_ad0582b1f784.slice/crio-06c548208a117748a043c1a60e314d394e6af948626db446fbd8fd84f4450872 WatchSource:0}: Error finding container 06c548208a117748a043c1a60e314d394e6af948626db446fbd8fd84f4450872: Status 404 returned error can't find the container with id 06c548208a117748a043c1a60e314d394e6af948626db446fbd8fd84f4450872 Nov 03 22:02:59.841225 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.841186 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:59.841335 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.841271 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:02:59.841335 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:02:59.841308 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:03:00.069572 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.069531 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" event={"ID":"15288224-fbef-47db-b0a5-8d4f488a6915","Type":"ContainerStarted","Data":"c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2"} Nov 03 22:03:00.070003 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.069598 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" event={"ID":"15288224-fbef-47db-b0a5-8d4f488a6915","Type":"ContainerStarted","Data":"ec97caa5a80acad381cd97f076f1ad8bddaa49260b7c64f06cf0c734b78e333a"} Nov 03 22:03:00.070573 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.070552 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:03:00.073006 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.072976 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" event={"ID":"06145d74-f6d7-4895-8e8e-ad0582b1f784","Type":"ContainerStarted","Data":"29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139"} Nov 03 22:03:00.073145 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.073036 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" event={"ID":"06145d74-f6d7-4895-8e8e-ad0582b1f784","Type":"ContainerStarted","Data":"06c548208a117748a043c1a60e314d394e6af948626db446fbd8fd84f4450872"} Nov 03 22:03:00.074884 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.074858 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" event={"ID":"85227cfc-1db8-46e0-964a-c24442389ad2","Type":"ContainerStarted","Data":"5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2"} Nov 03 22:03:00.075018 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.074910 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" event={"ID":"85227cfc-1db8-46e0-964a-c24442389ad2","Type":"ContainerStarted","Data":"3b65b5ead7b9116bcc3b8004af28eb3000824f24bf974a98ab2cbb176b112bfc"} Nov 03 22:03:00.075018 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.074943 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:03:00.186777 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.186717 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" podStartSLOduration=2.186698698 podStartE2EDuration="2.186698698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:00.183264728 +0000 UTC m=+2847.794112544" watchObservedRunningTime="2026-11-03 22:03:00.186698698 +0000 UTC m=+2847.797546518" Nov 03 22:03:00.300641 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.300549 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" podStartSLOduration=2.300528557 podStartE2EDuration="2.300528557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:00.295880467 +0000 UTC m=+2847.906728282" watchObservedRunningTime="2026-11-03 22:03:00.300528557 +0000 UTC m=+2847.911376372" Nov 03 22:03:00.400760 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.400732 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:03:00.406558 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.406534 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:03:00.413763 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:00.413711 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" podStartSLOduration=1.413696826 podStartE2EDuration="1.413696826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:00.409500474 +0000 UTC m=+2848.020348314" watchObservedRunningTime="2026-11-03 22:03:00.413696826 +0000 UTC m=+2848.024544640" Nov 03 22:03:01.081587 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:01.081544 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:03:01.082808 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:01.082779 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:03:03.069738 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:03:03.069693 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:03:03.224221 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:03.224187 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:03.470742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:03.470711 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:03.702070 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:03.702041 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:04.120514 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:04.120484 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr"] Nov 03 22:03:04.225212 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:04.225171 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml"] Nov 03 22:03:04.572418 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:04.572332 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:04.800858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:04.800819 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:08.375904 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:08.375866 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:09.121371 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.121337 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" podUID="ab4743a3-60ad-4f7c-82a8-f72018ef061f" containerName="payload-processing" containerID="cri-o://ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6" gracePeriod=25 Nov 03 22:03:09.225721 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.225685 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" podUID="b252e860-5da3-4066-89c9-e575c6c66d2f" containerName="payload-pre-processing" containerID="cri-o://143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101" gracePeriod=25 Nov 03 22:03:09.388357 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.388292 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 22:03:09.501653 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.501108 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 22:03:09.506301 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.506274 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4743a3-60ad-4f7c-82a8-f72018ef061f-kube-api-access-wjcdv\" (UniqueName: \"kubernetes.io/projected/ab4743a3-60ad-4f7c-82a8-f72018ef061f-kube-api-access-wjcdv\") pod \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " Nov 03 22:03:09.506530 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.506514 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4743a3-60ad-4f7c-82a8-f72018ef061f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4743a3-60ad-4f7c-82a8-f72018ef061f-plugins-config-volume\") pod \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\" (UID: \"ab4743a3-60ad-4f7c-82a8-f72018ef061f\") " Nov 03 22:03:09.506873 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.506853 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4743a3-60ad-4f7c-82a8-f72018ef061f-plugins-config-volume" pod "ab4743a3-60ad-4f7c-82a8-f72018ef061f" (UID: "ab4743a3-60ad-4f7c-82a8-f72018ef061f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:09.507028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.507015 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4743a3-60ad-4f7c-82a8-f72018ef061f-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:09.508508 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.508481 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4743a3-60ad-4f7c-82a8-f72018ef061f-kube-api-access-wjcdv" pod "ab4743a3-60ad-4f7c-82a8-f72018ef061f" (UID: "ab4743a3-60ad-4f7c-82a8-f72018ef061f"). InnerVolumeSpecName "kube-api-access-wjcdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:09.608375 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.608338 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b252e860-5da3-4066-89c9-e575c6c66d2f-kube-api-access-qp8vn\" (UniqueName: \"kubernetes.io/projected/b252e860-5da3-4066-89c9-e575c6c66d2f-kube-api-access-qp8vn\") pod \"b252e860-5da3-4066-89c9-e575c6c66d2f\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " Nov 03 22:03:09.608375 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.608376 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b252e860-5da3-4066-89c9-e575c6c66d2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b252e860-5da3-4066-89c9-e575c6c66d2f-plugins-config-volume\") pod \"b252e860-5da3-4066-89c9-e575c6c66d2f\" (UID: \"b252e860-5da3-4066-89c9-e575c6c66d2f\") " Nov 03 22:03:09.608837 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.608808 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b252e860-5da3-4066-89c9-e575c6c66d2f-plugins-config-volume" pod "b252e860-5da3-4066-89c9-e575c6c66d2f" (UID: "b252e860-5da3-4066-89c9-e575c6c66d2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:09.609160 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.609139 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjcdv\" (UniqueName: \"kubernetes.io/projected/ab4743a3-60ad-4f7c-82a8-f72018ef061f-kube-api-access-wjcdv\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:09.609160 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.609160 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b252e860-5da3-4066-89c9-e575c6c66d2f-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:09.610686 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.610655 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b252e860-5da3-4066-89c9-e575c6c66d2f-kube-api-access-qp8vn" pod "b252e860-5da3-4066-89c9-e575c6c66d2f" (UID: "b252e860-5da3-4066-89c9-e575c6c66d2f"). InnerVolumeSpecName "kube-api-access-qp8vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:09.710726 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:09.710602 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8vn\" (UniqueName: \"kubernetes.io/projected/b252e860-5da3-4066-89c9-e575c6c66d2f-kube-api-access-qp8vn\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:10.143884 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.143840 5931 generic.go:356] "Generic (PLEG): container finished" podID="b252e860-5da3-4066-89c9-e575c6c66d2f" containerID="143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101" exitCode=0 Nov 03 22:03:10.144038 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.143906 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" Nov 03 22:03:10.144038 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.143908 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" event={"ID":"b252e860-5da3-4066-89c9-e575c6c66d2f","Type":"ContainerDied","Data":"143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101"} Nov 03 22:03:10.144038 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.144006 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml" event={"ID":"b252e860-5da3-4066-89c9-e575c6c66d2f","Type":"ContainerDied","Data":"644c3abf34135266c3dade1a5682ddbaf280ca1eeda7a44ef46baff74727237f"} Nov 03 22:03:10.144038 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.144037 5931 scope.go:122] "RemoveContainer" containerID="143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101" Nov 03 22:03:10.153493 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.153463 5931 generic.go:356] "Generic (PLEG): container finished" podID="ab4743a3-60ad-4f7c-82a8-f72018ef061f" containerID="ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6" exitCode=0 Nov 03 22:03:10.153598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.153549 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" event={"ID":"ab4743a3-60ad-4f7c-82a8-f72018ef061f","Type":"ContainerDied","Data":"ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6"} Nov 03 22:03:10.153704 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.153597 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" event={"ID":"ab4743a3-60ad-4f7c-82a8-f72018ef061f","Type":"ContainerDied","Data":"dbe05209f0ed8d83e7bb840b71783085e126b21bfce28dcd3c41cb91e4b4ea6d"} Nov 03 22:03:10.153759 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.153708 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr" Nov 03 22:03:10.167369 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.167347 5931 scope.go:122] "RemoveContainer" containerID="143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101" Nov 03 22:03:10.167813 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:03:10.167794 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101\": container with ID starting with 143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101 not found: ID does not exist" containerID="143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101" Nov 03 22:03:10.167874 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.167823 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101"} err="failed to get container status \"143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101\": rpc error: code = NotFound desc = could not find container \"143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101\": container with ID starting with 143cd650c61e12bcf2ac09e8c3d3b2138be17d3fc09b72d37a3bf5c1417bf101 not found: ID does not exist" Nov 03 22:03:10.167874 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.167840 5931 scope.go:122] "RemoveContainer" containerID="ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6" Nov 03 22:03:10.176834 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.176814 5931 scope.go:122] "RemoveContainer" containerID="ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6" Nov 03 22:03:10.177097 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:03:10.177078 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6\": container with ID starting with ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6 not found: ID does not exist" containerID="ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6" Nov 03 22:03:10.177152 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.177111 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6"} err="failed to get container status \"ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6\": rpc error: code = NotFound desc = could not find container \"ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6\": container with ID starting with ec296783ffa66b0953e54975524ca0543acbd3ec6ef160d381a189395dc8c2e6 not found: ID does not exist" Nov 03 22:03:10.309283 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.309237 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr"] Nov 03 22:03:10.358928 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.358891 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7990bd-b5c4dcfd4-wpqnr"] Nov 03 22:03:10.420217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.420140 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml"] Nov 03 22:03:10.426279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.426255 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7990bd-dfb55cb7f-6dqml"] Nov 03 22:03:10.506835 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.506805 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4743a3-60ad-4f7c-82a8-f72018ef061f" path="/var/lib/kubelet/pods/ab4743a3-60ad-4f7c-82a8-f72018ef061f/volumes" Nov 03 22:03:10.507250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:10.507235 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b252e860-5da3-4066-89c9-e575c6c66d2f" path="/var/lib/kubelet/pods/b252e860-5da3-4066-89c9-e575c6c66d2f/volumes" Nov 03 22:03:11.084210 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:11.084180 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:03:11.756959 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:11.756927 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:11.995203 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:11.995175 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:12.090843 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:12.090820 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:03:13.378521 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:13.378488 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:17.617658 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:17.617603 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:17.845318 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:17.845276 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:18.120143 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:18.120115 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:22.818511 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:22.818432 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:27.588999 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:27.588948 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:31.622527 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:31.622487 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:33.070040 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:03:33.070005 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:03:33.146103 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:33.146080 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:33.380489 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:33.380456 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:33.601674 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:33.601647 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:34.090688 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:34.090663 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:36.020311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:36.020276 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:38.432084 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:38.432027 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:39.912879 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:39.912852 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:40.182499 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:40.182417 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:40.417109 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:40.417077 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:44.098176 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:44.098082 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:44.329637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:44.329564 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:44.702762 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:44.702731 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:44.926782 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:44.926740 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:47.404694 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.404659 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt"] Nov 03 22:03:47.405129 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.404837 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" podUID="e717b7f8-9269-4af4-be82-ba7a38ee5ca8" containerName="maas-api" containerID="cri-o://fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8" gracePeriod=30 Nov 03 22:03:47.603210 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.603181 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:47.685055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.685030 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 22:03:47.767214 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.767185 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-openshift-service-ca\") pod \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " Nov 03 22:03:47.767358 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.767221 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-kube-api-access-2w57t\" (UniqueName: \"kubernetes.io/projected/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-kube-api-access-2w57t\") pod \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " Nov 03 22:03:47.767358 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.767280 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-metrics-certs\") pod \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " Nov 03 22:03:47.767358 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.767338 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls\") pod \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\" (UID: \"e717b7f8-9269-4af4-be82-ba7a38ee5ca8\") " Nov 03 22:03:47.767526 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.767500 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-openshift-service-ca" pod "e717b7f8-9269-4af4-be82-ba7a38ee5ca8" (UID: "e717b7f8-9269-4af4-be82-ba7a38ee5ca8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:47.767967 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.767900 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:47.769487 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.769461 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-metrics-certs" pod "e717b7f8-9269-4af4-be82-ba7a38ee5ca8" (UID: "e717b7f8-9269-4af4-be82-ba7a38ee5ca8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:47.769633 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.769534 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls" pod "e717b7f8-9269-4af4-be82-ba7a38ee5ca8" (UID: "e717b7f8-9269-4af4-be82-ba7a38ee5ca8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:47.769633 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.769540 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-kube-api-access-2w57t" pod "e717b7f8-9269-4af4-be82-ba7a38ee5ca8" (UID: "e717b7f8-9269-4af4-be82-ba7a38ee5ca8"). InnerVolumeSpecName "kube-api-access-2w57t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:47.869208 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.869177 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w57t\" (UniqueName: \"kubernetes.io/projected/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-kube-api-access-2w57t\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:47.869208 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.869206 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:47.869386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:47.869216 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e717b7f8-9269-4af4-be82-ba7a38ee5ca8-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:48.400873 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.400838 5931 generic.go:356] "Generic (PLEG): container finished" podID="e717b7f8-9269-4af4-be82-ba7a38ee5ca8" containerID="fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8" exitCode=0 Nov 03 22:03:48.401064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.400894 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" Nov 03 22:03:48.401064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.400898 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" event={"ID":"e717b7f8-9269-4af4-be82-ba7a38ee5ca8","Type":"ContainerDied","Data":"fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8"} Nov 03 22:03:48.401064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.401050 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt" event={"ID":"e717b7f8-9269-4af4-be82-ba7a38ee5ca8","Type":"ContainerDied","Data":"db2ff3032aa7727f8c8e211af497c1b4c536e726656c58697d7809bd2bc29ff8"} Nov 03 22:03:48.401290 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.401081 5931 scope.go:122] "RemoveContainer" containerID="fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8" Nov 03 22:03:48.438651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.432861 5931 scope.go:122] "RemoveContainer" containerID="fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8" Nov 03 22:03:48.438651 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:03:48.437217 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8\": container with ID starting with fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8 not found: ID does not exist" containerID="fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8" Nov 03 22:03:48.438651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.437264 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8"} err="failed to get container status \"fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8\": rpc error: code = NotFound desc = could not find container \"fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8\": container with ID starting with fd36bc2b6901f28b3cdb0e3c90ec905c951166c4be2e75369131219ba946d6d8 not found: ID does not exist" Nov 03 22:03:48.564468 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.564437 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt"] Nov 03 22:03:48.567168 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.567141 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb3520-7ddb9fcd7b-wmpqt"] Nov 03 22:03:48.618375 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.618346 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z"] Nov 03 22:03:48.618604 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.618581 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" podUID="6e549544-2c68-4acb-b081-8132b43b127e" containerName="maas-api" containerID="cri-o://3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a" gracePeriod=30 Nov 03 22:03:48.810934 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.810886 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:48.883134 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.883107 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:03:48.981644 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.981551 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-metrics-certs\") pod \"6e549544-2c68-4acb-b081-8132b43b127e\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " Nov 03 22:03:48.981780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.981653 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e549544-2c68-4acb-b081-8132b43b127e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e549544-2c68-4acb-b081-8132b43b127e-openshift-service-ca\") pod \"6e549544-2c68-4acb-b081-8132b43b127e\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " Nov 03 22:03:48.981780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.981745 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls\") pod \"6e549544-2c68-4acb-b081-8132b43b127e\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " Nov 03 22:03:48.981901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.981867 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e549544-2c68-4acb-b081-8132b43b127e-kube-api-access-gblbr\" (UniqueName: \"kubernetes.io/projected/6e549544-2c68-4acb-b081-8132b43b127e-kube-api-access-gblbr\") pod \"6e549544-2c68-4acb-b081-8132b43b127e\" (UID: \"6e549544-2c68-4acb-b081-8132b43b127e\") " Nov 03 22:03:48.981980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.981963 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e549544-2c68-4acb-b081-8132b43b127e-openshift-service-ca" pod "6e549544-2c68-4acb-b081-8132b43b127e" (UID: "6e549544-2c68-4acb-b081-8132b43b127e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:48.982354 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.982323 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e549544-2c68-4acb-b081-8132b43b127e-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:48.983860 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.983828 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls" pod "6e549544-2c68-4acb-b081-8132b43b127e" (UID: "6e549544-2c68-4acb-b081-8132b43b127e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:48.983975 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.983931 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-metrics-certs" pod "6e549544-2c68-4acb-b081-8132b43b127e" (UID: "6e549544-2c68-4acb-b081-8132b43b127e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:48.983975 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:48.983932 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e549544-2c68-4acb-b081-8132b43b127e-kube-api-access-gblbr" pod "6e549544-2c68-4acb-b081-8132b43b127e" (UID: "6e549544-2c68-4acb-b081-8132b43b127e"). InnerVolumeSpecName "kube-api-access-gblbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:49.083701 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.083669 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:49.083701 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.083697 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblbr\" (UniqueName: \"kubernetes.io/projected/6e549544-2c68-4acb-b081-8132b43b127e-kube-api-access-gblbr\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:49.083701 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.083707 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e549544-2c68-4acb-b081-8132b43b127e-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:03:49.408276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.408242 5931 generic.go:356] "Generic (PLEG): container finished" podID="6e549544-2c68-4acb-b081-8132b43b127e" containerID="3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a" exitCode=0 Nov 03 22:03:49.408462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.408286 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" Nov 03 22:03:49.408462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.408322 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" event={"ID":"6e549544-2c68-4acb-b081-8132b43b127e","Type":"ContainerDied","Data":"3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a"} Nov 03 22:03:49.408462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.408369 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z" event={"ID":"6e549544-2c68-4acb-b081-8132b43b127e","Type":"ContainerDied","Data":"e19cda067fe2ed2a188a7ae981cfe0e4127cbfa43db14ca25d1bd87cd81d740a"} Nov 03 22:03:49.408462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.408389 5931 scope.go:122] "RemoveContainer" containerID="3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a" Nov 03 22:03:49.419381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.419230 5931 scope.go:122] "RemoveContainer" containerID="3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a" Nov 03 22:03:49.419493 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:03:49.419473 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a\": container with ID starting with 3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a not found: ID does not exist" containerID="3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a" Nov 03 22:03:49.419548 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.419501 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a"} err="failed to get container status \"3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a\": rpc error: code = NotFound desc = could not find container \"3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a\": container with ID starting with 3fac826e5cdd958b7c4a7f983fc4c3e05f70438d5d590775c3987c22ab3df62a not found: ID does not exist" Nov 03 22:03:49.571827 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.571803 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z"] Nov 03 22:03:49.575574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.575555 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7990bd-7bc557f88f-8tv5z"] Nov 03 22:03:49.583019 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.582989 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:49.917004 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:49.916958 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:50.505661 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:50.505596 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e549544-2c68-4acb-b081-8132b43b127e" path="/var/lib/kubelet/pods/6e549544-2c68-4acb-b081-8132b43b127e/volumes" Nov 03 22:03:50.506041 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:50.506026 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e717b7f8-9269-4af4-be82-ba7a38ee5ca8" path="/var/lib/kubelet/pods/e717b7f8-9269-4af4-be82-ba7a38ee5ca8/volumes" Nov 03 22:03:50.602978 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:50.602940 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:50.859696 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:50.859657 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:51.087493 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:51.087462 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:51.705676 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:51.705597 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:52.412507 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:52.412477 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:55.467866 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:55.467834 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5"] Nov 03 22:03:55.630091 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:55.630061 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:55.990658 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:55.990627 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:57.480231 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:57.480200 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:57.720153 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:57.720121 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:58.031334 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:58.031292 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:58.340247 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:58.340215 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:58.566901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:58.566872 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:58.967532 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:58.967504 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:59.552018 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.551984 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8"] Nov 03 22:03:59.552645 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552602 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b252e860-5da3-4066-89c9-e575c6c66d2f" containerName="payload-pre-processing" Nov 03 22:03:59.552645 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552638 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b252e860-5da3-4066-89c9-e575c6c66d2f" containerName="payload-pre-processing" Nov 03 22:03:59.552804 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552657 5931 cpu_manager.go:414] "containerMap: removing container" podUID="ab4743a3-60ad-4f7c-82a8-f72018ef061f" containerName="payload-processing" Nov 03 22:03:59.552804 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552664 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4743a3-60ad-4f7c-82a8-f72018ef061f" containerName="payload-processing" Nov 03 22:03:59.552804 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552680 5931 cpu_manager.go:414] "containerMap: removing container" podUID="e717b7f8-9269-4af4-be82-ba7a38ee5ca8" containerName="maas-api" Nov 03 22:03:59.552804 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552685 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e717b7f8-9269-4af4-be82-ba7a38ee5ca8" containerName="maas-api" Nov 03 22:03:59.552804 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552710 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6e549544-2c68-4acb-b081-8132b43b127e" containerName="maas-api" Nov 03 22:03:59.552804 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552717 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e549544-2c68-4acb-b081-8132b43b127e" containerName="maas-api" Nov 03 22:03:59.552988 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552816 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b252e860-5da3-4066-89c9-e575c6c66d2f" containerName="payload-pre-processing" Nov 03 22:03:59.552988 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552827 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4743a3-60ad-4f7c-82a8-f72018ef061f" containerName="payload-processing" Nov 03 22:03:59.552988 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552835 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e717b7f8-9269-4af4-be82-ba7a38ee5ca8" containerName="maas-api" Nov 03 22:03:59.552988 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.552844 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e549544-2c68-4acb-b081-8132b43b127e" containerName="maas-api" Nov 03 22:03:59.556181 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.556166 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.644892 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.644866 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e15bbc-openshift-default-dockercfg-8tmjx\"" Nov 03 22:03:59.660983 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.660958 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8"] Nov 03 22:03:59.695192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695159 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206bb57e-8add-4c82-be28-d47efff96d09-istio-podinfo\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695351 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695298 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-credential-socket\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695423 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695389 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-socket\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695549 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695526 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhlln\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-kube-api-access-rhlln\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695592 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206bb57e-8add-4c82-be28-d47efff96d09-istiod-ca-cert\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695639 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-istio-token\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695675 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-data\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695722 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-envoy\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.695805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.695768 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-certs\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.742624 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.742578 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:59.797432 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.797395 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206bb57e-8add-4c82-be28-d47efff96d09-istiod-ca-cert\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.797432 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.797436 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-istio-token\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.797738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.797462 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-data\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.797738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.797593 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-envoy\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.797738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.797706 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-certs\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.797918 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.797742 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206bb57e-8add-4c82-be28-d47efff96d09-istio-podinfo\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.797918 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.797866 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-data\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.798096 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.798077 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-certs\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.798242 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.798223 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-credential-socket\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.798492 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.798251 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206bb57e-8add-4c82-be28-d47efff96d09-istiod-ca-cert\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.798492 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.798282 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-socket\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.798492 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.798383 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhlln\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-kube-api-access-rhlln\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.798492 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.798483 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-credential-socket\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.798778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.798752 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-socket\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.800121 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.800099 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-envoy\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.800662 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.800603 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206bb57e-8add-4c82-be28-d47efff96d09-istio-podinfo\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.854302 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.854276 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-istio-token\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.855078 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.855044 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhlln\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-kube-api-access-rhlln\") pod \"e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.869562 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.869537 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:03:59.971012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:03:59.970984 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:00.301024 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.301001 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8"] Nov 03 22:04:00.303034 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:04:00.303007 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206bb57e_8add_4c82_be28_d47efff96d09.slice/crio-fb466785d78799749c802b981ea34d2380699c38a7885370c125393931593135 WatchSource:0}: Error finding container fb466785d78799749c802b981ea34d2380699c38a7885370c125393931593135: Status 404 returned error can't find the container with id fb466785d78799749c802b981ea34d2380699c38a7885370c125393931593135 Nov 03 22:04:00.305213 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.305181 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:04:00.305277 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.305246 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:04:00.305312 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.305274 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:04:00.488761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.488730 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" event={"ID":"206bb57e-8add-4c82-be28-d47efff96d09","Type":"ContainerStarted","Data":"1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c"} Nov 03 22:04:00.488929 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.488779 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" event={"ID":"206bb57e-8add-4c82-be28-d47efff96d09","Type":"ContainerStarted","Data":"fb466785d78799749c802b981ea34d2380699c38a7885370c125393931593135"} Nov 03 22:04:00.629323 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.629277 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" podStartSLOduration=1.62926424 podStartE2EDuration="1.62926424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:04:00.626139118 +0000 UTC m=+2908.236986932" watchObservedRunningTime="2026-11-03 22:04:00.62926424 +0000 UTC m=+2908.240112054" Nov 03 22:04:00.678694 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.678660 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:00.870030 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.870006 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:04:00.906362 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:00.906271 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:01.875571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:01.875542 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:04:02.528097 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:02.528056 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:04:02.528975 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:02.528957 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:04:03.071587 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:03.071517 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:04:03.696800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:03.696771 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:03.933923 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:03.933895 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:05.889726 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:05.889693 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:08.958089 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:08.958048 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:09.203730 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:09.203695 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:10.490415 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.490380 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" podUID="ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" containerName="main" containerID="cri-o://2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11" gracePeriod=45 Nov 03 22:04:10.746026 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.745960 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 22:04:10.814498 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814469 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-home\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-home\") pod \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " Nov 03 22:04:10.814674 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814576 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tmp-dir\") pod \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " Nov 03 22:04:10.814739 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814688 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-dshm\") pod \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " Nov 03 22:04:10.814739 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814731 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-model-cache\") pod \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " Nov 03 22:04:10.814857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814780 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-home" pod "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" (UID: "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:10.814857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814806 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tls-certs\") pod \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " Nov 03 22:04:10.814857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814834 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-kube-api-access-dcflx\" (UniqueName: \"kubernetes.io/projected/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-kube-api-access-dcflx\") pod \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\" (UID: \"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a\") " Nov 03 22:04:10.814972 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814850 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tmp-dir" pod "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" (UID: "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:10.815010 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.814984 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-model-cache" pod "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" (UID: "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:10.815279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.815233 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:10.815279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.815250 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:10.815279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.815258 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:10.816990 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.816969 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-dshm" pod "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" (UID: "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:10.817231 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.817218 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tls-certs" pod "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" (UID: "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:10.818801 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.818774 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-kube-api-access-dcflx" pod "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" (UID: "ec04a1f5-a1b6-4604-a27f-633b30bf3e6a"). InnerVolumeSpecName "kube-api-access-dcflx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:10.916047 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.916019 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:10.916047 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.916047 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:10.916258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:10.916062 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcflx\" (UniqueName: \"kubernetes.io/projected/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a-kube-api-access-dcflx\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:11.031021 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.030859 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:11.584312 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.584280 5931 generic.go:356] "Generic (PLEG): container finished" podID="ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" containerID="2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11" exitCode=0 Nov 03 22:04:11.584858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.584315 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" Nov 03 22:04:11.584858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.584331 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" event={"ID":"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a","Type":"ContainerDied","Data":"2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11"} Nov 03 22:04:11.584858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.584354 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5" event={"ID":"ec04a1f5-a1b6-4604-a27f-633b30bf3e6a","Type":"ContainerDied","Data":"686f2bfec2e6821e29434f0b0e4113d22018323ae37fa9cc236d655e62d81f60"} Nov 03 22:04:11.584858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.584371 5931 scope.go:122] "RemoveContainer" containerID="2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11" Nov 03 22:04:11.596349 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.596335 5931 scope.go:122] "RemoveContainer" containerID="2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11" Nov 03 22:04:11.596630 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:11.596598 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11\": container with ID starting with 2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11 not found: ID does not exist" containerID="2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11" Nov 03 22:04:11.596700 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.596643 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11"} err="failed to get container status \"2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11\": rpc error: code = NotFound desc = could not find container \"2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11\": container with ID starting with 2c343ae13daf465c30df235fa854fbe2a435a782c1fb394995233bf4335cba11 not found: ID does not exist" Nov 03 22:04:11.748218 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.748189 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5"] Nov 03 22:04:11.797775 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:11.797749 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-eb3520/praxis-route-eb3520-kserve-7b99659bc5-vj6b5"] Nov 03 22:04:12.505460 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:12.505430 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" path="/var/lib/kubelet/pods/ec04a1f5-a1b6-4604-a27f-633b30bf3e6a/volumes" Nov 03 22:04:15.885923 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:15.885890 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:17.068360 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:17.068329 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:20.680178 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:20.680144 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:25.440077 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:25.440034 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:25.703901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:25.703823 5931 generic.go:356] "Generic (PLEG): container finished" podID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerID="29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139" exitCode=0 Nov 03 22:04:25.704064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:25.703892 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" event={"ID":"06145d74-f6d7-4895-8e8e-ad0582b1f784","Type":"ContainerDied","Data":"29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139"} Nov 03 22:04:25.704283 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:25.704268 5931 scope.go:122] "RemoveContainer" containerID="29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139" Nov 03 22:04:25.705916 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:25.705888 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:04:25.705992 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:25.705972 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:04:25.706035 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:25.706015 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:04:26.712045 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:26.712007 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" event={"ID":"06145d74-f6d7-4895-8e8e-ad0582b1f784","Type":"ContainerStarted","Data":"d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322"} Nov 03 22:04:27.401020 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:27.400984 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:04:27.402210 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:27.402188 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:04:27.402345 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:27.402235 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:04:28.400884 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:28.400848 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:04:28.401250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:28.400898 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:04:29.400565 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:29.400535 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:04:29.400779 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:29.400592 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:04:30.163606 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:30.163571 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:30.400546 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:30.400512 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:04:30.400752 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:30.400581 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:04:31.081326 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:31.081281 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:04:31.400458 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:31.400419 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:04:31.400882 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:31.400481 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:04:31.665139 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:31.665049 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:32.405064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:32.405032 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:04:32.406170 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:32.406149 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:04:33.069036 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:33.068997 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:04:33.267034 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:33.267004 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:34.893080 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:34.893040 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:38.938150 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:38.938117 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:39.185258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:39.185221 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:39.415011 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:39.414979 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:40.712485 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:40.712448 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:40.969689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:40.969596 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:41.188765 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:41.188738 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:41.708707 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:41.708681 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:46.021410 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:46.021370 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:47.797836 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:47.797781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:48.037958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:48.037924 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:48.057032 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:48.056957 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98"] Nov 03 22:04:48.057197 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:48.057175 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" containerID="cri-o://19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97" gracePeriod=30 Nov 03 22:04:48.295790 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:48.295762 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:48.605962 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:48.605933 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:49.053455 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:49.053385 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:49.670449 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:49.670418 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:49.908747 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:49.908704 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:50.519317 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.519280 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6"] Nov 03 22:04:50.519816 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.519491 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" podUID="6fa25de6-4fd2-4a96-9205-32d2d02b5642" containerName="maas-api" containerID="cri-o://06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417" gracePeriod=30 Nov 03 22:04:50.737679 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.737577 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:50.787422 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.787341 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 22:04:50.886268 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.886240 5931 generic.go:356] "Generic (PLEG): container finished" podID="6fa25de6-4fd2-4a96-9205-32d2d02b5642" containerID="06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417" exitCode=0 Nov 03 22:04:50.886404 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.886286 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" Nov 03 22:04:50.886404 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.886324 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" event={"ID":"6fa25de6-4fd2-4a96-9205-32d2d02b5642","Type":"ContainerDied","Data":"06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417"} Nov 03 22:04:50.886404 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.886375 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6" event={"ID":"6fa25de6-4fd2-4a96-9205-32d2d02b5642","Type":"ContainerDied","Data":"98b31cbf59756172f0e87e741f7928fff3c39ba77b99e7d1f2923c98c7bee052"} Nov 03 22:04:50.886404 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.886398 5931 scope.go:122] "RemoveContainer" containerID="06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417" Nov 03 22:04:50.893503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.893482 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa25de6-4fd2-4a96-9205-32d2d02b5642-kube-api-access-v9mqc\" (UniqueName: \"kubernetes.io/projected/6fa25de6-4fd2-4a96-9205-32d2d02b5642-kube-api-access-v9mqc\") pod \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " Nov 03 22:04:50.893645 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.893552 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls\") pod \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " Nov 03 22:04:50.893718 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.893700 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa25de6-4fd2-4a96-9205-32d2d02b5642-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa25de6-4fd2-4a96-9205-32d2d02b5642-openshift-service-ca\") pod \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " Nov 03 22:04:50.895051 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.893828 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-metrics-certs\") pod \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\" (UID: \"6fa25de6-4fd2-4a96-9205-32d2d02b5642\") " Nov 03 22:04:50.895051 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.894366 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa25de6-4fd2-4a96-9205-32d2d02b5642-openshift-service-ca" pod "6fa25de6-4fd2-4a96-9205-32d2d02b5642" (UID: "6fa25de6-4fd2-4a96-9205-32d2d02b5642"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:50.895545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.895522 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa25de6-4fd2-4a96-9205-32d2d02b5642-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:50.919063 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.918994 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls" pod "6fa25de6-4fd2-4a96-9205-32d2d02b5642" (UID: "6fa25de6-4fd2-4a96-9205-32d2d02b5642"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:50.919650 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.919585 5931 scope.go:122] "RemoveContainer" containerID="06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417" Nov 03 22:04:50.920167 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:50.920097 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417\": container with ID starting with 06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417 not found: ID does not exist" containerID="06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417" Nov 03 22:04:50.920167 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.920136 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417"} err="failed to get container status \"06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417\": rpc error: code = NotFound desc = could not find container \"06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417\": container with ID starting with 06a916f5021f2ee7fd14554fffc5f45bffb040e304b2cbb5c736577140ee9417 not found: ID does not exist" Nov 03 22:04:50.921999 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.921972 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-metrics-certs" pod "6fa25de6-4fd2-4a96-9205-32d2d02b5642" (UID: "6fa25de6-4fd2-4a96-9205-32d2d02b5642"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:50.923089 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:50.923042 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa25de6-4fd2-4a96-9205-32d2d02b5642-kube-api-access-v9mqc" pod "6fa25de6-4fd2-4a96-9205-32d2d02b5642" (UID: "6fa25de6-4fd2-4a96-9205-32d2d02b5642"). InnerVolumeSpecName "kube-api-access-v9mqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:51.001945 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.001917 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:51.001945 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.001944 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9mqc\" (UniqueName: \"kubernetes.io/projected/6fa25de6-4fd2-4a96-9205-32d2d02b5642-kube-api-access-v9mqc\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:51.002137 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.001954 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa25de6-4fd2-4a96-9205-32d2d02b5642-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:51.013054 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.013023 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:51.058120 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.058051 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5"] Nov 03 22:04:51.163839 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.163815 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg"] Nov 03 22:04:51.350176 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.350143 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:51.395870 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.395837 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6"] Nov 03 22:04:51.398454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.398435 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-95d604-cb7fcd598-gnkp6"] Nov 03 22:04:51.572714 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.572689 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:51.642017 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.641958 5931 patch_prober.go:29] interesting pod/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:04:51.642017 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:51.642001 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:04:52.507074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:52.507041 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa25de6-4fd2-4a96-9205-32d2d02b5642" path="/var/lib/kubelet/pods/6fa25de6-4fd2-4a96-9205-32d2d02b5642/volumes" Nov 03 22:04:52.736289 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:52.736257 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:52.984667 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:52.984638 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:53.324002 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.323976 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:04:53.427752 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.427726 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-envoy\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.427863 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.427786 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b030ce95-ff95-417d-b111-c1fbdba2c301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b030ce95-ff95-417d-b111-c1fbdba2c301-istiod-ca-cert\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.427863 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.427829 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-token\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-token\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.427952 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.427873 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-kube-api-access-48pd7\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-kube-api-access-48pd7\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.427992 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.427950 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-data\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.427992 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.427982 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-podinfo\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.428074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428007 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-credential-socket\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.428074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428040 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-certs\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.428181 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428108 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-socket\") pod \"b030ce95-ff95-417d-b111-c1fbdba2c301\" (UID: \"b030ce95-ff95-417d-b111-c1fbdba2c301\") " Nov 03 22:04:53.428237 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428209 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-data" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:53.428305 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428247 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b030ce95-ff95-417d-b111-c1fbdba2c301-istiod-ca-cert" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:53.428557 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428446 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-socket" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:53.428557 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428534 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-credential-socket" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:53.428707 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428660 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-certs" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:53.428870 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428849 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.428974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428877 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b030ce95-ff95-417d-b111-c1fbdba2c301-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.428974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428892 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.428974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428906 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.428974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.428921 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.429942 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.429923 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-envoy" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:53.430087 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.430069 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-token" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:53.430197 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.430182 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-kube-api-access-48pd7" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "kube-api-access-48pd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:53.430498 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.430483 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-podinfo" pod "b030ce95-ff95-417d-b111-c1fbdba2c301" (UID: "b030ce95-ff95-417d-b111-c1fbdba2c301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:04:53.529472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.529408 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.529472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.529435 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.529472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.529451 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48pd7\" (UniqueName: \"kubernetes.io/projected/b030ce95-ff95-417d-b111-c1fbdba2c301-kube-api-access-48pd7\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.529472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.529464 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b030ce95-ff95-417d-b111-c1fbdba2c301-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:53.915853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.915823 5931 generic.go:356] "Generic (PLEG): container finished" podID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerID="19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97" exitCode=0 Nov 03 22:04:53.916315 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.915885 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" Nov 03 22:04:53.916315 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.915907 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" event={"ID":"b030ce95-ff95-417d-b111-c1fbdba2c301","Type":"ContainerDied","Data":"19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97"} Nov 03 22:04:53.916315 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.915962 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98" event={"ID":"b030ce95-ff95-417d-b111-c1fbdba2c301","Type":"ContainerDied","Data":"fcc16b0aa9bb1e25a418ae01371012684f1ff703972442c57be5645952c8781f"} Nov 03 22:04:53.916315 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.915990 5931 scope.go:122] "RemoveContainer" containerID="19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97" Nov 03 22:04:53.928314 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.928298 5931 scope.go:122] "RemoveContainer" containerID="362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259" Nov 03 22:04:53.938147 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.938134 5931 scope.go:122] "RemoveContainer" containerID="19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97" Nov 03 22:04:53.938365 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:53.938350 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97\": container with ID starting with 19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97 not found: ID does not exist" containerID="19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97" Nov 03 22:04:53.938408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.938374 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97"} err="failed to get container status \"19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97\": rpc error: code = NotFound desc = could not find container \"19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97\": container with ID starting with 19cb7dad369ef945781317f1de6964e352856c2dc80ad993a1d16b943b8a7d97 not found: ID does not exist" Nov 03 22:04:53.938408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.938390 5931 scope.go:122] "RemoveContainer" containerID="362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259" Nov 03 22:04:53.938604 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:53.938590 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259\": container with ID starting with 362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259 not found: ID does not exist" containerID="362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259" Nov 03 22:04:53.938687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:53.938637 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259"} err="failed to get container status \"362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259\": rpc error: code = NotFound desc = could not find container \"362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259\": container with ID starting with 362dfceb1863accf72ab61cf8de49c34081732f89b58f70e430a4229dff7d259 not found: ID does not exist" Nov 03 22:04:54.080490 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:54.080467 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98"] Nov 03 22:04:54.131215 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:54.131191 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bd35c-openshift-default-77f878f54f-qmb98"] Nov 03 22:04:54.431753 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:54.431731 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:54.505958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:54.505925 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" path="/var/lib/kubelet/pods/b030ce95-ff95-417d-b111-c1fbdba2c301/volumes" Nov 03 22:04:54.668590 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:54.668556 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:55.189773 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:55.189726 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:55.520743 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:55.520661 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:56.059209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.059176 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" podUID="0d460f72-809b-4ba4-a50e-b667d1b29dce" containerName="payload-processing" containerID="cri-o://ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501" gracePeriod=25 Nov 03 22:04:56.163885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.163859 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:56.164696 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.164668 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" podUID="1482d9f0-ed14-4549-80ce-4d04364ab142" containerName="payload-pre-processing" containerID="cri-o://f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6" gracePeriod=25 Nov 03 22:04:56.330272 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.330202 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 22:04:56.437069 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.437044 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 22:04:56.458457 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.458430 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d460f72-809b-4ba4-a50e-b667d1b29dce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d460f72-809b-4ba4-a50e-b667d1b29dce-plugins-config-volume\") pod \"0d460f72-809b-4ba4-a50e-b667d1b29dce\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " Nov 03 22:04:56.458593 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.458553 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d460f72-809b-4ba4-a50e-b667d1b29dce-kube-api-access-xsxst\" (UniqueName: \"kubernetes.io/projected/0d460f72-809b-4ba4-a50e-b667d1b29dce-kube-api-access-xsxst\") pod \"0d460f72-809b-4ba4-a50e-b667d1b29dce\" (UID: \"0d460f72-809b-4ba4-a50e-b667d1b29dce\") " Nov 03 22:04:56.458903 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.458878 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d460f72-809b-4ba4-a50e-b667d1b29dce-plugins-config-volume" pod "0d460f72-809b-4ba4-a50e-b667d1b29dce" (UID: "0d460f72-809b-4ba4-a50e-b667d1b29dce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:56.459204 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.459178 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d460f72-809b-4ba4-a50e-b667d1b29dce-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:56.461028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.460995 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d460f72-809b-4ba4-a50e-b667d1b29dce-kube-api-access-xsxst" pod "0d460f72-809b-4ba4-a50e-b667d1b29dce" (UID: "0d460f72-809b-4ba4-a50e-b667d1b29dce"). InnerVolumeSpecName "kube-api-access-xsxst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:56.563564 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.560021 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1482d9f0-ed14-4549-80ce-4d04364ab142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1482d9f0-ed14-4549-80ce-4d04364ab142-plugins-config-volume\") pod \"1482d9f0-ed14-4549-80ce-4d04364ab142\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " Nov 03 22:04:56.563564 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.560194 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1482d9f0-ed14-4549-80ce-4d04364ab142-kube-api-access-jvqrd\" (UniqueName: \"kubernetes.io/projected/1482d9f0-ed14-4549-80ce-4d04364ab142-kube-api-access-jvqrd\") pod \"1482d9f0-ed14-4549-80ce-4d04364ab142\" (UID: \"1482d9f0-ed14-4549-80ce-4d04364ab142\") " Nov 03 22:04:56.563564 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.563220 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxst\" (UniqueName: \"kubernetes.io/projected/0d460f72-809b-4ba4-a50e-b667d1b29dce-kube-api-access-xsxst\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:56.565830 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.564185 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1482d9f0-ed14-4549-80ce-4d04364ab142-plugins-config-volume" pod "1482d9f0-ed14-4549-80ce-4d04364ab142" (UID: "1482d9f0-ed14-4549-80ce-4d04364ab142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:56.573243 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.573211 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1482d9f0-ed14-4549-80ce-4d04364ab142-kube-api-access-jvqrd" pod "1482d9f0-ed14-4549-80ce-4d04364ab142" (UID: "1482d9f0-ed14-4549-80ce-4d04364ab142"). InnerVolumeSpecName "kube-api-access-jvqrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:56.664125 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.664101 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1482d9f0-ed14-4549-80ce-4d04364ab142-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:56.664264 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.664129 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqrd\" (UniqueName: \"kubernetes.io/projected/1482d9f0-ed14-4549-80ce-4d04364ab142-kube-api-access-jvqrd\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:04:56.938331 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.938253 5931 generic.go:356] "Generic (PLEG): container finished" podID="1482d9f0-ed14-4549-80ce-4d04364ab142" containerID="f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6" exitCode=0 Nov 03 22:04:56.938331 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.938318 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" Nov 03 22:04:56.938479 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.938337 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" event={"ID":"1482d9f0-ed14-4549-80ce-4d04364ab142","Type":"ContainerDied","Data":"f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6"} Nov 03 22:04:56.938479 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.938422 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg" event={"ID":"1482d9f0-ed14-4549-80ce-4d04364ab142","Type":"ContainerDied","Data":"fd0874c64114bc893427c3b109deb667f69dc16099484e1f9b37c607c5a4ccde"} Nov 03 22:04:56.938479 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.938446 5931 scope.go:122] "RemoveContainer" containerID="f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6" Nov 03 22:04:56.939780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.939752 5931 generic.go:356] "Generic (PLEG): container finished" podID="0d460f72-809b-4ba4-a50e-b667d1b29dce" containerID="ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501" exitCode=0 Nov 03 22:04:56.939910 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.939793 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" event={"ID":"0d460f72-809b-4ba4-a50e-b667d1b29dce","Type":"ContainerDied","Data":"ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501"} Nov 03 22:04:56.939910 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.939820 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" Nov 03 22:04:56.939910 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.939850 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5" event={"ID":"0d460f72-809b-4ba4-a50e-b667d1b29dce","Type":"ContainerDied","Data":"e1069695f6d4c0958ac3e306f23ec48299345fd82e5a2f1261f9a18fa7ef396e"} Nov 03 22:04:56.947986 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.947972 5931 scope.go:122] "RemoveContainer" containerID="f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6" Nov 03 22:04:56.948239 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:56.948221 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6\": container with ID starting with f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6 not found: ID does not exist" containerID="f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6" Nov 03 22:04:56.948299 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.948245 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6"} err="failed to get container status \"f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6\": rpc error: code = NotFound desc = could not find container \"f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6\": container with ID starting with f7a0dc5d661f4367214ea0fe6e94728eeac57cd7ce08793a79b5e742b7bb7ef6 not found: ID does not exist" Nov 03 22:04:56.948299 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.948261 5931 scope.go:122] "RemoveContainer" containerID="ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501" Nov 03 22:04:56.956690 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.956675 5931 scope.go:122] "RemoveContainer" containerID="ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501" Nov 03 22:04:56.956970 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:56.956948 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501\": container with ID starting with ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501 not found: ID does not exist" containerID="ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501" Nov 03 22:04:56.957037 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:56.956979 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501"} err="failed to get container status \"ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501\": rpc error: code = NotFound desc = could not find container \"ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501\": container with ID starting with ad0d7734110c8eb821dbd342c72636e65ada90f606c6f129d3f5711ebee15501 not found: ID does not exist" Nov 03 22:04:57.108194 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.108165 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5"] Nov 03 22:04:57.157365 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.157331 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-95d604-8577b86c8b-bmcc5"] Nov 03 22:04:57.220866 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.220797 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg"] Nov 03 22:04:57.226460 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.226432 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-95d604-6ccd8945bc-zqmqg"] Nov 03 22:04:57.436775 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.436750 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58"] Nov 03 22:04:57.437301 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437287 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6fa25de6-4fd2-4a96-9205-32d2d02b5642" containerName="maas-api" Nov 03 22:04:57.437344 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437303 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa25de6-4fd2-4a96-9205-32d2d02b5642" containerName="maas-api" Nov 03 22:04:57.437344 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437316 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0d460f72-809b-4ba4-a50e-b667d1b29dce" containerName="payload-processing" Nov 03 22:04:57.437344 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437320 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d460f72-809b-4ba4-a50e-b667d1b29dce" containerName="payload-processing" Nov 03 22:04:57.437344 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437341 5931 cpu_manager.go:414] "containerMap: removing container" podUID="ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" containerName="main" Nov 03 22:04:57.437471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437349 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" containerName="main" Nov 03 22:04:57.437471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437366 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" Nov 03 22:04:57.437471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437371 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" Nov 03 22:04:57.437471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437381 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" Nov 03 22:04:57.437471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437385 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" Nov 03 22:04:57.437471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437395 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1482d9f0-ed14-4549-80ce-4d04364ab142" containerName="payload-pre-processing" Nov 03 22:04:57.437471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437399 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1482d9f0-ed14-4549-80ce-4d04364ab142" containerName="payload-pre-processing" Nov 03 22:04:57.437719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437482 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" Nov 03 22:04:57.437719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437492 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b030ce95-ff95-417d-b111-c1fbdba2c301" containerName="istio-proxy" Nov 03 22:04:57.437719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437500 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1482d9f0-ed14-4549-80ce-4d04364ab142" containerName="payload-pre-processing" Nov 03 22:04:57.437719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437514 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa25de6-4fd2-4a96-9205-32d2d02b5642" containerName="maas-api" Nov 03 22:04:57.437719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437521 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d460f72-809b-4ba4-a50e-b667d1b29dce" containerName="payload-processing" Nov 03 22:04:57.437719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.437530 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec04a1f5-a1b6-4604-a27f-633b30bf3e6a" containerName="main" Nov 03 22:04:57.442215 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.442197 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.494230 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.494170 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e15bbc-serving-cert\"" Nov 03 22:04:57.498417 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.498396 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58"] Nov 03 22:04:57.575065 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.575038 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b97c4-9953-4b74-acbe-01ecf1043be4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.575213 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.575106 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6x9k\" (UniqueName: \"kubernetes.io/projected/8a6b97c4-9953-4b74-acbe-01ecf1043be4-kube-api-access-v6x9k\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.575213 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.575186 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.575328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.575273 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.676410 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.676380 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.676571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.676497 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.676657 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.676625 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b97c4-9953-4b74-acbe-01ecf1043be4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.676657 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:57.676644 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-serving-cert: secret "maas-api-e2e-shared-a-w3-e15bbc-serving-cert" not found Nov 03 22:04:57.676744 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:04:57.676721 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls podName:8a6b97c4-9953-4b74-acbe-01ecf1043be4 nodeName:}" failed. No retries permitted until 2026-11-03 22:04:58.176699309 +0000 UTC m=+2965.787547117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls") pod "maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" (UID: "8a6b97c4-9953-4b74-acbe-01ecf1043be4") : secret "maas-api-e2e-shared-a-w3-e15bbc-serving-cert" not found Nov 03 22:04:57.676808 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.676742 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6x9k\" (UniqueName: \"kubernetes.io/projected/8a6b97c4-9953-4b74-acbe-01ecf1043be4-kube-api-access-v6x9k\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.677247 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.677227 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b97c4-9953-4b74-acbe-01ecf1043be4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.679058 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.679041 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.740064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.739138 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6x9k\" (UniqueName: \"kubernetes.io/projected/8a6b97c4-9953-4b74-acbe-01ecf1043be4-kube-api-access-v6x9k\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:57.743355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:57.743330 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:58.034003 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.033976 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:58.181871 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.181838 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:58.184346 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.184320 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:58.265794 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.265762 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:58.268791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.268758 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd"] Nov 03 22:04:58.273965 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.273941 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.290988 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.290917 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd"] Nov 03 22:04:58.325967 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.325941 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-0d45ba\"" Nov 03 22:04:58.325967 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.325960 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-0d45ba-dockercfg-kj2z8\"" Nov 03 22:04:58.355805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.355787 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:58.384178 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.384153 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-kube-api-access-8kvwr\") pod \"payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.384370 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.384345 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.407485 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.407457 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4"] Nov 03 22:04:58.414767 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.413824 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:58.485794 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.485762 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-kube-api-access-8kvwr\") pod \"payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.486160 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.485913 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.486537 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.486519 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.506725 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.506697 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d460f72-809b-4ba4-a50e-b667d1b29dce" path="/var/lib/kubelet/pods/0d460f72-809b-4ba4-a50e-b667d1b29dce/volumes" Nov 03 22:04:58.507167 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.507147 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1482d9f0-ed14-4549-80ce-4d04364ab142" path="/var/lib/kubelet/pods/1482d9f0-ed14-4549-80ce-4d04364ab142/volumes" Nov 03 22:04:58.507478 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.507460 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4"] Nov 03 22:04:58.541581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.541528 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-kube-api-access-8kvwr\") pod \"payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.587064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.587045 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f06984-5f02-4e02-97fb-6d63c69fca38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:58.587215 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.587195 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtht\" (UniqueName: \"kubernetes.io/projected/09f06984-5f02-4e02-97fb-6d63c69fca38-kube-api-access-zgtht\") pod \"payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:58.600639 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.600589 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:58.688578 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.688532 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtht\" (UniqueName: \"kubernetes.io/projected/09f06984-5f02-4e02-97fb-6d63c69fca38-kube-api-access-zgtht\") pod \"payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:58.688735 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.688707 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f06984-5f02-4e02-97fb-6d63c69fca38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:58.690267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.690197 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f06984-5f02-4e02-97fb-6d63c69fca38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:58.733075 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.733055 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58"] Nov 03 22:04:58.735212 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:04:58.735188 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6b97c4_9953_4b74_acbe_01ecf1043be4.slice/crio-4aa556246dbea0bc3137c0c9d5e5474ccc10d61a4742493ed5163896ac60a7b9 WatchSource:0}: Error finding container 4aa556246dbea0bc3137c0c9d5e5474ccc10d61a4742493ed5163896ac60a7b9: Status 404 returned error can't find the container with id 4aa556246dbea0bc3137c0c9d5e5474ccc10d61a4742493ed5163896ac60a7b9 Nov 03 22:04:58.744855 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.744832 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtht\" (UniqueName: \"kubernetes.io/projected/09f06984-5f02-4e02-97fb-6d63c69fca38-kube-api-access-zgtht\") pod \"payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:58.957671 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.957648 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" event={"ID":"8a6b97c4-9953-4b74-acbe-01ecf1043be4","Type":"ContainerStarted","Data":"4aa556246dbea0bc3137c0c9d5e5474ccc10d61a4742493ed5163896ac60a7b9"} Nov 03 22:04:58.975237 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:58.975212 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd"] Nov 03 22:04:58.976869 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:04:58.976845 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ca03b3_6b7c_4160_a9a1_9ebf98009784.slice/crio-9407b531e36f956fe9a09396fbdc4aa55ab4c163430a9ea1d67d8a4ff551cf91 WatchSource:0}: Error finding container 9407b531e36f956fe9a09396fbdc4aa55ab4c163430a9ea1d67d8a4ff551cf91: Status 404 returned error can't find the container with id 9407b531e36f956fe9a09396fbdc4aa55ab4c163430a9ea1d67d8a4ff551cf91 Nov 03 22:04:59.035736 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.035714 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:59.408856 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.408829 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4"] Nov 03 22:04:59.409907 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:04:59.409883 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f06984_5f02_4e02_97fb_6d63c69fca38.slice/crio-819e9bb2e5fa5bc17b4df94c84299cfebf494ad3245765ea22674239ba055e6b WatchSource:0}: Error finding container 819e9bb2e5fa5bc17b4df94c84299cfebf494ad3245765ea22674239ba055e6b: Status 404 returned error can't find the container with id 819e9bb2e5fa5bc17b4df94c84299cfebf494ad3245765ea22674239ba055e6b Nov 03 22:04:59.965358 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.965322 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" event={"ID":"8a6b97c4-9953-4b74-acbe-01ecf1043be4","Type":"ContainerStarted","Data":"bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3"} Nov 03 22:04:59.965845 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.965417 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:04:59.967040 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.967014 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" event={"ID":"09f06984-5f02-4e02-97fb-6d63c69fca38","Type":"ContainerStarted","Data":"734b816f1128833468d6505991a4a0e6ee7914a3883bc425e892c2cd8d9c0205"} Nov 03 22:04:59.967167 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.967059 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" event={"ID":"09f06984-5f02-4e02-97fb-6d63c69fca38","Type":"ContainerStarted","Data":"819e9bb2e5fa5bc17b4df94c84299cfebf494ad3245765ea22674239ba055e6b"} Nov 03 22:04:59.967167 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.967100 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:04:59.968604 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.968581 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" event={"ID":"d2ca03b3-6b7c-4160-a9a1-9ebf98009784","Type":"ContainerStarted","Data":"a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967"} Nov 03 22:04:59.968780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.968660 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:04:59.968780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:04:59.968684 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" event={"ID":"d2ca03b3-6b7c-4160-a9a1-9ebf98009784","Type":"ContainerStarted","Data":"9407b531e36f956fe9a09396fbdc4aa55ab4c163430a9ea1d67d8a4ff551cf91"} Nov 03 22:05:00.077959 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:00.077911 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" podStartSLOduration=2.267502143 podStartE2EDuration="3.077893019s" totalImagesPullingTime="810.390876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:00.076768365 +0000 UTC m=+2967.687616180" watchObservedRunningTime="2026-11-03 22:05:00.077893019 +0000 UTC m=+2967.688740824" Nov 03 22:05:00.190199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:00.190152 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" podStartSLOduration=2.190139842 podStartE2EDuration="2.190139842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:00.187755019 +0000 UTC m=+2967.798602833" watchObservedRunningTime="2026-11-03 22:05:00.190139842 +0000 UTC m=+2967.800987698" Nov 03 22:05:00.299461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:00.298900 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" podStartSLOduration=2.298878964 podStartE2EDuration="2.298878964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:00.298535267 +0000 UTC m=+2967.909383082" watchObservedRunningTime="2026-11-03 22:05:00.298878964 +0000 UTC m=+2967.909726780" Nov 03 22:05:00.989378 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:00.989336 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:01.482058 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:01.482024 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:01.723355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:01.723310 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:01.886889 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:01.886860 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55"] Nov 03 22:05:01.890847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:01.890823 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:01.943180 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:01.942734 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-75f577-openshift-default-dockercfg-qt5wg\"" Nov 03 22:05:02.002636 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:01.996502 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55"] Nov 03 22:05:02.030273 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030242 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-credential-socket\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030281 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-envoy\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030379 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8nbh\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-kube-api-access-v8nbh\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030421 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-socket\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030448 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-token\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030599 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-podinfo\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030736 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030720 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c27741f-5733-4eae-92a1-40eb8fc93976-istiod-ca-cert\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030864 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030849 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-certs\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.030927 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.030912 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-data\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.081438 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.081411 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:02.131828 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.131788 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-credential-socket\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.131828 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.131833 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-envoy\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132106 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.131869 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8nbh\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-kube-api-access-v8nbh\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132106 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.131895 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-socket\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132106 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.131917 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-token\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132106 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.131991 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-podinfo\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132106 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132031 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c27741f-5733-4eae-92a1-40eb8fc93976-istiod-ca-cert\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132364 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132119 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-certs\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132364 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132180 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-data\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132364 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132251 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-credential-socket\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132514 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132485 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-data\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132582 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132537 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-certs\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132655 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132582 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-socket\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.132830 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.132812 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c27741f-5733-4eae-92a1-40eb8fc93976-istiod-ca-cert\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.134500 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.134479 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-envoy\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.135026 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.135006 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-podinfo\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.191565 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.191491 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-token\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.192074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.192052 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8nbh\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-kube-api-access-v8nbh\") pod \"e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.203744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.203717 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:02.475898 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.475822 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:02.646498 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.646466 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55"] Nov 03 22:05:02.679963 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:05:02.679936 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c27741f_5733_4eae_92a1_40eb8fc93976.slice/crio-24cf47b213977e76b2b1349d2762a0ad85904a65f3e67792672d81bac4166353 WatchSource:0}: Error finding container 24cf47b213977e76b2b1349d2762a0ad85904a65f3e67792672d81bac4166353: Status 404 returned error can't find the container with id 24cf47b213977e76b2b1349d2762a0ad85904a65f3e67792672d81bac4166353 Nov 03 22:05:02.682530 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.682493 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:05:02.682660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.682561 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:05:02.682660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.682590 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:05:02.758698 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:02.758668 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:03.008495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:03.008385 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" event={"ID":"1c27741f-5733-4eae-92a1-40eb8fc93976","Type":"ContainerStarted","Data":"3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39"} Nov 03 22:05:03.008495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:03.008447 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" event={"ID":"1c27741f-5733-4eae-92a1-40eb8fc93976","Type":"ContainerStarted","Data":"24cf47b213977e76b2b1349d2762a0ad85904a65f3e67792672d81bac4166353"} Nov 03 22:05:03.020670 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:03.020638 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:03.109047 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:05:03.109011 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:05:03.130866 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:03.130813 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podStartSLOduration=2.130795159 podStartE2EDuration="2.130795159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:03.129751271 +0000 UTC m=+2970.740599085" watchObservedRunningTime="2026-11-03 22:05:03.130795159 +0000 UTC m=+2970.741642976" Nov 03 22:05:03.204693 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:03.204649 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:03.210633 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:03.210598 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:03.250203 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:03.249998 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:04.014601 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:04.014575 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:04.015385 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:04.015365 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:05:05.979558 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:05.979531 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:05:06.110605 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:06.110554 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:06.349454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:06.349412 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:08.014561 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:08.014525 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:10.977563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:10.977525 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:05:10.977970 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:10.977599 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:05:12.112267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.112234 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:12.341830 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.341787 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:12.422820 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.422746 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t"] Nov 03 22:05:12.426919 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.426894 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.528726 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.528691 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-75f577\"" Nov 03 22:05:12.529127 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.529052 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-75f577-dockercfg-cnfkx\"" Nov 03 22:05:12.532427 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.532406 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t"] Nov 03 22:05:12.572077 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.572042 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.572260 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.572224 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgdc\" (UniqueName: \"kubernetes.io/projected/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-kube-api-access-9wgdc\") pod \"payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.586259 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.586230 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7"] Nov 03 22:05:12.590791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.590244 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.626234 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.626211 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7"] Nov 03 22:05:12.673746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.673671 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8pt\" (UniqueName: \"kubernetes.io/projected/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-kube-api-access-xl8pt\") pod \"payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.673746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.673723 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.673958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.673931 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.674065 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.674049 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgdc\" (UniqueName: \"kubernetes.io/projected/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-kube-api-access-9wgdc\") pod \"payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.674602 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.674585 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.730200 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.730175 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgdc\" (UniqueName: \"kubernetes.io/projected/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-kube-api-access-9wgdc\") pod \"payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.740305 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.740274 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:12.775054 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.775026 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8pt\" (UniqueName: \"kubernetes.io/projected/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-kube-api-access-xl8pt\") pod \"payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.775187 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.775077 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.775995 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.775972 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.831365 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.831329 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8pt\" (UniqueName: \"kubernetes.io/projected/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-kube-api-access-xl8pt\") pod \"payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.914325 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.914292 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:12.948131 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:12.948101 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:13.050651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:13.049189 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd"] Nov 03 22:05:13.050651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:13.049469 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abeee6e5-dvdgd"] Nov 03 22:05:13.160688 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:13.160663 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t"] Nov 03 22:05:13.162190 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:05:13.162166 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0118c7bd_d838_4fb3_bb2a_f6f6aad0be9f.slice/crio-3f9301e17f68f910aa3eecf91fb873e28f370cfab9c3cc4b0db2a16d2d3a6114 WatchSource:0}: Error finding container 3f9301e17f68f910aa3eecf91fb873e28f370cfab9c3cc4b0db2a16d2d3a6114: Status 404 returned error can't find the container with id 3f9301e17f68f910aa3eecf91fb873e28f370cfab9c3cc4b0db2a16d2d3a6114 Nov 03 22:05:13.293733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:13.293705 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7"] Nov 03 22:05:13.295229 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:05:13.295198 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dbaa83_a7a0_4b87_8e91_cb0b02c5e67a.slice/crio-3467d314ff303a4f6f14c2436f33469078f7b85571eafa385d4d6dc3f8974a86 WatchSource:0}: Error finding container 3467d314ff303a4f6f14c2436f33469078f7b85571eafa385d4d6dc3f8974a86: Status 404 returned error can't find the container with id 3467d314ff303a4f6f14c2436f33469078f7b85571eafa385d4d6dc3f8974a86 Nov 03 22:05:14.082018 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.081965 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" event={"ID":"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a","Type":"ContainerStarted","Data":"d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d"} Nov 03 22:05:14.082218 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.082036 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" event={"ID":"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a","Type":"ContainerStarted","Data":"3467d314ff303a4f6f14c2436f33469078f7b85571eafa385d4d6dc3f8974a86"} Nov 03 22:05:14.082218 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.082068 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:14.083852 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.083822 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" event={"ID":"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f","Type":"ContainerStarted","Data":"4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f"} Nov 03 22:05:14.083992 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.083870 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" event={"ID":"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f","Type":"ContainerStarted","Data":"3f9301e17f68f910aa3eecf91fb873e28f370cfab9c3cc4b0db2a16d2d3a6114"} Nov 03 22:05:14.083992 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.083947 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:14.196503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.196448 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" podStartSLOduration=2.19643518 podStartE2EDuration="2.19643518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:14.193782388 +0000 UTC m=+2981.804630204" watchObservedRunningTime="2026-11-03 22:05:14.19643518 +0000 UTC m=+2981.807282994" Nov 03 22:05:14.325405 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.325346 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" podStartSLOduration=2.32532659 podStartE2EDuration="2.32532659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:14.320426506 +0000 UTC m=+2981.931274331" watchObservedRunningTime="2026-11-03 22:05:14.32532659 +0000 UTC m=+2981.936174406" Nov 03 22:05:14.506413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:14.506382 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4577cd3-9b4a-4712-8929-33664251dc2f" path="/var/lib/kubelet/pods/b4577cd3-9b4a-4712-8929-33664251dc2f/volumes" Nov 03 22:05:17.652696 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:17.651457 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:22.658715 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:22.658683 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:22.936187 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:22.936117 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:24.177439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:24.177369 5931 generic.go:356] "Generic (PLEG): container finished" podID="206bb57e-8add-4c82-be28-d47efff96d09" containerID="1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c" exitCode=0 Nov 03 22:05:24.177824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:24.177438 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" event={"ID":"206bb57e-8add-4c82-be28-d47efff96d09","Type":"ContainerDied","Data":"1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c"} Nov 03 22:05:24.177824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:24.177777 5931 scope.go:122] "RemoveContainer" containerID="1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c" Nov 03 22:05:24.179074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:24.179049 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:05:24.179132 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:24.179115 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:05:24.179172 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:24.179146 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:05:25.093050 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:25.093018 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:05:25.093296 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:25.093276 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:05:25.185578 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:25.185545 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" event={"ID":"206bb57e-8add-4c82-be28-d47efff96d09","Type":"ContainerStarted","Data":"612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7"} Nov 03 22:05:25.870159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:25.870126 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:05:25.870348 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:25.870279 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:05:25.870348 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:25.870322 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:05:26.870455 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:26.870423 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:05:26.870837 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:26.870474 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:05:27.337307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:27.337208 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:27.870040 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:27.870008 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:05:27.870226 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:27.870061 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:05:28.870062 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:28.870029 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:05:28.870456 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:28.870093 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:05:29.869831 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:29.869799 5931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:05:29.870005 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:29.869858 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:05:30.874765 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:30.874737 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:05:30.875192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:30.875175 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:05:30.875792 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:30.875774 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:05:32.057195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:32.057137 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:32.584497 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:32.584471 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:33.068875 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:05:33.068844 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:05:33.405025 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:33.404966 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:33.670135 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:33.670062 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:33.952780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:33.952679 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:35.736275 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:35.736245 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:05:35.742225 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:35.742206 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:05:36.904510 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:36.904474 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:37.784435 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:37.784406 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:38.031595 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:38.031560 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:41.071199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.071170 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:41.306416 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.306382 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:41.708472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.708438 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd"] Nov 03 22:05:41.712169 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.712151 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:41.768651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.768627 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd"] Nov 03 22:05:41.813686 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.813663 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-27a3c3\"" Nov 03 22:05:41.813892 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.813872 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-27a3c3-dockercfg-8tkdn\"" Nov 03 22:05:41.829126 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.829099 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fqv\" (UniqueName: \"kubernetes.io/projected/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-kube-api-access-86fqv\") pod \"payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:41.829333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.829314 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:41.930318 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.930294 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:41.930447 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.930346 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fqv\" (UniqueName: \"kubernetes.io/projected/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-kube-api-access-86fqv\") pod \"payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:41.930990 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.930967 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:41.986970 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:41.986921 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fqv\" (UniqueName: \"kubernetes.io/projected/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-kube-api-access-86fqv\") pod \"payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:42.025355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:42.025329 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:42.277349 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:42.277269 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:42.400236 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:42.400188 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd"] Nov 03 22:05:42.403494 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:05:42.403467 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f3a47b_f391_4ba3_ae39_c5add2a38fe0.slice/crio-0ff499e105239881c8cfcf80a78784c9a4b9874e2cab97397973296dc229117a WatchSource:0}: Error finding container 0ff499e105239881c8cfcf80a78784c9a4b9874e2cab97397973296dc229117a: Status 404 returned error can't find the container with id 0ff499e105239881c8cfcf80a78784c9a4b9874e2cab97397973296dc229117a Nov 03 22:05:43.301760 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:43.301724 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" event={"ID":"69f3a47b-f391-4ba3-ae39-c5add2a38fe0","Type":"ContainerStarted","Data":"8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6"} Nov 03 22:05:43.302250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:43.301781 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" event={"ID":"69f3a47b-f391-4ba3-ae39-c5add2a38fe0","Type":"ContainerStarted","Data":"0ff499e105239881c8cfcf80a78784c9a4b9874e2cab97397973296dc229117a"} Nov 03 22:05:43.302250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:43.301865 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:43.418785 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:43.418729 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" podStartSLOduration=2.418712924 podStartE2EDuration="2.418712924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:43.416826407 +0000 UTC m=+3011.027674221" watchObservedRunningTime="2026-11-03 22:05:43.418712924 +0000 UTC m=+3011.029560738" Nov 03 22:05:47.157841 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:47.157808 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:47.371241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:47.371201 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:52.148749 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:52.148707 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:53.108636 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:53.105193 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:54.310516 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:54.310492 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:05:56.715132 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:05:56.715090 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:01.443465 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:01.443433 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:03.068572 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:06:03.068519 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:06:06.435328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.435293 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6"] Nov 03 22:06:06.439644 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.439624 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.445031 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.445007 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:06.491108 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.491059 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6v6\" (UniqueName: \"kubernetes.io/projected/3823ed9d-12a2-4b8f-81cf-691926b28a43-kube-api-access-xh6v6\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.491307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.491110 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-dshm\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.491307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.491224 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3823ed9d-12a2-4b8f-81cf-691926b28a43-tls-certs\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.491453 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.491401 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-tmp-dir\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.491513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.491432 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-home\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.491513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.491478 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-model-cache\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.496037 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.496006 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6"] Nov 03 22:06:06.508581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.508548 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"openshift-service-ca.crt\"" Nov 03 22:06:06.538602 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.538573 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"kube-root-ca.crt\"" Nov 03 22:06:06.541423 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.541406 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"default-dockercfg-x5nkn\"" Nov 03 22:06:06.541530 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.541436 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ddbba\"/\"auth-test-model-6ddbba-kserve-self-signed-certs\"" Nov 03 22:06:06.592433 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592413 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-tmp-dir\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.592552 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592444 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-home\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.592552 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592462 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-model-cache\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.592704 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592685 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6v6\" (UniqueName: \"kubernetes.io/projected/3823ed9d-12a2-4b8f-81cf-691926b28a43-kube-api-access-xh6v6\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.592921 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592900 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-home\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.593055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592948 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-tmp-dir\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.593055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592966 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-model-cache\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.593055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.592953 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-dshm\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.593241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.593089 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3823ed9d-12a2-4b8f-81cf-691926b28a43-tls-certs\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.595241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.595221 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-dshm\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.595571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.595550 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3823ed9d-12a2-4b8f-81cf-691926b28a43-tls-certs\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.649762 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.649740 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6v6\" (UniqueName: \"kubernetes.io/projected/3823ed9d-12a2-4b8f-81cf-691926b28a43-kube-api-access-xh6v6\") pod \"auth-test-model-6ddbba-kserve-54b569dddf-967s6\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.664710 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.664689 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:06.753229 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.753173 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:06.886735 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:06.886685 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:07.105652 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.105626 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:07.129886 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.129862 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6"] Nov 03 22:06:07.131605 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:06:07.131581 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3823ed9d_12a2_4b8f_81cf_691926b28a43.slice/crio-4cba611c950d16151197624e25a0752529be15e393478f7a50ec013f9f2820a4 WatchSource:0}: Error finding container 4cba611c950d16151197624e25a0752529be15e393478f7a50ec013f9f2820a4: Status 404 returned error can't find the container with id 4cba611c950d16151197624e25a0752529be15e393478f7a50ec013f9f2820a4 Nov 03 22:06:07.489281 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.488187 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" event={"ID":"3823ed9d-12a2-4b8f-81cf-691926b28a43","Type":"ContainerStarted","Data":"6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308"} Nov 03 22:06:07.489281 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.488238 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" event={"ID":"3823ed9d-12a2-4b8f-81cf-691926b28a43","Type":"ContainerStarted","Data":"4cba611c950d16151197624e25a0752529be15e393478f7a50ec013f9f2820a4"} Nov 03 22:06:07.579318 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.579278 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:07.609969 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.609918 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" podStartSLOduration=1.6099036770000001 podStartE2EDuration="1.609903677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:07.603782466 +0000 UTC m=+3035.214630281" watchObservedRunningTime="2026-11-03 22:06:07.609903677 +0000 UTC m=+3035.220751529" Nov 03 22:06:07.754019 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.753922 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:07.975968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:07.975931 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:11.976067 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:11.976031 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:12.792005 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:12.791973 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:13.011575 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:13.011548 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:16.753687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:16.753657 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:16.766362 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:16.766336 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:17.048786 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:17.048710 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:17.564946 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:17.564917 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:06:18.803478 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:18.803447 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:19.030715 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.030680 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:19.129865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.129840 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx"] Nov 03 22:06:19.134055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.134041 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.189703 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.189673 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx"] Nov 03 22:06:19.246633 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.240859 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7fb477\"" Nov 03 22:06:19.246633 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.241053 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7fb477-dockercfg-r6n26\"" Nov 03 22:06:19.269309 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.269269 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglvn\" (UniqueName: \"kubernetes.io/projected/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-kube-api-access-hglvn\") pod \"payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.269506 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.269462 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.285027 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.284998 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk"] Nov 03 22:06:19.289922 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.289897 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:19.342797 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.342767 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk"] Nov 03 22:06:19.370791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.370733 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.370963 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.370912 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglvn\" (UniqueName: \"kubernetes.io/projected/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-kube-api-access-hglvn\") pod \"payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.371391 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.371374 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.440914 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.440857 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglvn\" (UniqueName: \"kubernetes.io/projected/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-kube-api-access-hglvn\") pod \"payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.451303 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.451284 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:19.471685 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.471657 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcndw\" (UniqueName: \"kubernetes.io/projected/bd483f46-2923-4f02-bfe1-6fe9ad32d374-kube-api-access-zcndw\") pod \"payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:19.471917 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.471897 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd483f46-2923-4f02-bfe1-6fe9ad32d374-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:19.572654 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.572607 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd483f46-2923-4f02-bfe1-6fe9ad32d374-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:19.572813 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.572693 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcndw\" (UniqueName: \"kubernetes.io/projected/bd483f46-2923-4f02-bfe1-6fe9ad32d374-kube-api-access-zcndw\") pod \"payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:19.573235 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.573216 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd483f46-2923-4f02-bfe1-6fe9ad32d374-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:19.629351 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.629323 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcndw\" (UniqueName: \"kubernetes.io/projected/bd483f46-2923-4f02-bfe1-6fe9ad32d374-kube-api-access-zcndw\") pod \"payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:19.824053 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.824026 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx"] Nov 03 22:06:19.825264 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:06:19.825241 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc6892b_cda8_479d_9c04_b5d16efd8b4a.slice/crio-55773835b71889103b62ad15bb8d7bb7d6eb3873260857dd3719ba5d4096a6fa WatchSource:0}: Error finding container 55773835b71889103b62ad15bb8d7bb7d6eb3873260857dd3719ba5d4096a6fa: Status 404 returned error can't find the container with id 55773835b71889103b62ad15bb8d7bb7d6eb3873260857dd3719ba5d4096a6fa Nov 03 22:06:19.907104 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:19.907083 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:20.281368 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.281342 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk"] Nov 03 22:06:20.283478 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:06:20.283446 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd483f46_2923_4f02_bfe1_6fe9ad32d374.slice/crio-4f43648cc5a611b23ba7381a55b063f2c41ca9b4b3c950ec1d7831a01dc1aa74 WatchSource:0}: Error finding container 4f43648cc5a611b23ba7381a55b063f2c41ca9b4b3c950ec1d7831a01dc1aa74: Status 404 returned error can't find the container with id 4f43648cc5a611b23ba7381a55b063f2c41ca9b4b3c950ec1d7831a01dc1aa74 Nov 03 22:06:20.576250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.576168 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" event={"ID":"bd483f46-2923-4f02-bfe1-6fe9ad32d374","Type":"ContainerStarted","Data":"26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b"} Nov 03 22:06:20.576250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.576229 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:20.576460 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.576255 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" event={"ID":"bd483f46-2923-4f02-bfe1-6fe9ad32d374","Type":"ContainerStarted","Data":"4f43648cc5a611b23ba7381a55b063f2c41ca9b4b3c950ec1d7831a01dc1aa74"} Nov 03 22:06:20.577855 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.577824 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" event={"ID":"ecc6892b-cda8-479d-9c04-b5d16efd8b4a","Type":"ContainerStarted","Data":"dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036"} Nov 03 22:06:20.577993 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.577873 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" event={"ID":"ecc6892b-cda8-479d-9c04-b5d16efd8b4a","Type":"ContainerStarted","Data":"55773835b71889103b62ad15bb8d7bb7d6eb3873260857dd3719ba5d4096a6fa"} Nov 03 22:06:20.577993 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.577898 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:20.688085 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.688036 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" podStartSLOduration=1.6880171929999999 podStartE2EDuration="1.688017193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:20.687719926 +0000 UTC m=+3048.298567739" watchObservedRunningTime="2026-11-03 22:06:20.688017193 +0000 UTC m=+3048.298865045" Nov 03 22:06:20.803146 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:20.803103 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" podStartSLOduration=1.803090439 podStartE2EDuration="1.803090439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:20.799689325 +0000 UTC m=+3048.410537139" watchObservedRunningTime="2026-11-03 22:06:20.803090439 +0000 UTC m=+3048.413938253" Nov 03 22:06:22.231090 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:22.231059 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:27.343634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:27.335187 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:31.587527 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:31.587497 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:06:31.588011 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:31.587571 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:06:32.227852 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:32.227804 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:33.069390 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:06:33.069342 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:06:33.995384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:33.995353 5931 scope.go:122] "RemoveContainer" containerID="85da31fa2a269b76f514bb81377eb8a4fed83f0980bc2d727eb2a71fe50fea3b" Nov 03 22:06:35.057714 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.057684 5931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:06:35.058102 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.057738 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:06:35.705922 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.705856 5931 generic.go:356] "Generic (PLEG): container finished" podID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerID="3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39" exitCode=0 Nov 03 22:06:35.706100 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.705934 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" event={"ID":"1c27741f-5733-4eae-92a1-40eb8fc93976","Type":"ContainerDied","Data":"3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39"} Nov 03 22:06:35.706314 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.706297 5931 scope.go:122] "RemoveContainer" containerID="3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39" Nov 03 22:06:35.707813 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.707785 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:06:35.707901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.707853 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:06:35.707901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:35.707894 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:06:36.713957 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:36.713921 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" event={"ID":"1c27741f-5733-4eae-92a1-40eb8fc93976","Type":"ContainerStarted","Data":"50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c"} Nov 03 22:06:36.918822 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:36.918740 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:37.204704 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:37.204675 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:06:37.204966 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:37.204941 5931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:06:37.205058 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:37.204997 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 22:06:38.204050 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:38.204017 5931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:06:38.204436 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:38.204070 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 22:06:39.204875 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:39.204840 5931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:06:39.205260 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:39.204894 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 22:06:40.204109 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:40.204080 5931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:06:40.204290 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:40.204150 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 22:06:41.204741 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:41.204703 5931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:06:41.205195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:41.204764 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 22:06:41.552380 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:41.552279 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:41.762452 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:41.762422 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:41.974910 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:41.974883 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:42.208575 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:42.208547 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:06:42.209055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:42.208981 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:06:42.209649 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:42.209602 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:06:46.673423 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:46.673384 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:51.374790 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:51.374752 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:53.440001 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:53.439922 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:56.137491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:06:56.137458 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:00.915232 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:00.915197 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:02.243012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:02.242970 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:03.069428 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:07:03.069395 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:07:06.256061 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:06.256031 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:11.056816 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:11.056781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:13.794984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:13.794946 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:15.763684 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:15.763636 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:20.700762 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:20.700720 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:20.919363 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:20.919328 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:21.136545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:21.136520 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:24.245031 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:24.244997 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:25.859033 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:25.858992 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:32.271657 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:32.271600 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:33.067937 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:07:33.067900 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:07:35.195084 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:35.195019 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:39.919427 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:39.919382 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.644047 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:44.644010 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:49.357505 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:49.357466 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:53.825028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:53.824985 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:54.062509 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:54.062410 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:58.845049 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:07:58.845010 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:01.135220 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:01.135183 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:03.068832 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:08:03.068797 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:08:03.861868 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:03.861836 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv"] Nov 03 22:08:03.862080 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:03.862035 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" containerID="cri-o://65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e" gracePeriod=30 Nov 03 22:08:03.897727 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:03.897699 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:03.910721 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:03.910695 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg"] Nov 03 22:08:03.910913 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:03.910894 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" containerID="cri-o://8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46" gracePeriod=30 Nov 03 22:08:08.522907 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:08.522869 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:08.700124 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:08.700087 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:08:08.700287 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:08.700133 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:08:09.167835 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.167810 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:08:09.217276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.217255 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:08:09.348484 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348452 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-data\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.348689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348537 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-socket\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.348689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348599 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1419cffe-7409-4586-85b7-a40ab774dffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1419cffe-7409-4586-85b7-a40ab774dffd-istio-podinfo\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.348689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348660 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-kube-api-access-qtm25\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-kube-api-access-qtm25\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.348885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348689 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-kube-api-access-tzfs6\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-kube-api-access-tzfs6\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.348885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348810 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-podinfo\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.348885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348834 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-socket" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.348885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348864 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1419cffe-7409-4586-85b7-a40ab774dffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1419cffe-7409-4586-85b7-a40ab774dffd-istiod-ca-cert\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.349113 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.348932 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-istio-token\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-istio-token\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.349113 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349003 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-socket\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.349113 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349071 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-data\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.349113 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349103 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istiod-ca-cert\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.349351 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349145 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-credential-socket\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.349351 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349174 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-token\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-token\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349723 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-credential-socket" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349869 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-data" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349888 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-socket" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349915 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1419cffe-7409-4586-85b7-a40ab774dffd-istiod-ca-cert" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349919 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istiod-ca-cert" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349977 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-data" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.349267 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-certs\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350075 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-certs\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350129 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-envoy\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350132 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-certs" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350163 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-credential-socket\") pod \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\" (UID: \"68b6e331-b678-49ae-bd5f-9c2dba97ece7\") " Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350198 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-envoy\") pod \"1419cffe-7409-4586-85b7-a40ab774dffd\" (UID: \"1419cffe-7409-4586-85b7-a40ab774dffd\") " Nov 03 22:08:09.350439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350339 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-certs" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350537 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-credential-socket" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350884 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350907 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350924 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1419cffe-7409-4586-85b7-a40ab774dffd-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350939 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.350992 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.351104 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.351133 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.351146 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.351159 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.351172 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.351915 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.351819 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1419cffe-7409-4586-85b7-a40ab774dffd-istio-podinfo" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:08:09.352139 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.352106 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-istio-token" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:09.352832 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.352781 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-kube-api-access-qtm25" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "kube-api-access-qtm25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:09.353159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.353129 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-podinfo" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:08:09.353368 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.353333 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-token" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:09.353901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.353882 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-envoy" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.355019 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.354996 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-envoy" pod "1419cffe-7409-4586-85b7-a40ab774dffd" (UID: "1419cffe-7409-4586-85b7-a40ab774dffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.355136 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.355018 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-kube-api-access-tzfs6" pod "68b6e331-b678-49ae-bd5f-9c2dba97ece7" (UID: "68b6e331-b678-49ae-bd5f-9c2dba97ece7"). InnerVolumeSpecName "kube-api-access-tzfs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:09.362109 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.362082 5931 generic.go:356] "Generic (PLEG): container finished" podID="1419cffe-7409-4586-85b7-a40ab774dffd" containerID="8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46" exitCode=0 Nov 03 22:08:09.362203 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.362181 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" Nov 03 22:08:09.362265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.362228 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" event={"ID":"1419cffe-7409-4586-85b7-a40ab774dffd","Type":"ContainerDied","Data":"8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46"} Nov 03 22:08:09.362327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.362298 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg" event={"ID":"1419cffe-7409-4586-85b7-a40ab774dffd","Type":"ContainerDied","Data":"88e352c1256ec893fe2541621d3e66a8f9de293e1639ceab3176df70f5e7d884"} Nov 03 22:08:09.362382 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.362331 5931 scope.go:122] "RemoveContainer" containerID="8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46" Nov 03 22:08:09.364095 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.364070 5931 generic.go:356] "Generic (PLEG): container finished" podID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerID="65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e" exitCode=0 Nov 03 22:08:09.364212 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.364152 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" event={"ID":"68b6e331-b678-49ae-bd5f-9c2dba97ece7","Type":"ContainerDied","Data":"65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e"} Nov 03 22:08:09.364212 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.364192 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" event={"ID":"68b6e331-b678-49ae-bd5f-9c2dba97ece7","Type":"ContainerDied","Data":"2f903efa596b7c16633e5e0299b09427684ce548fc9feb7b321e4f4a2178ac28"} Nov 03 22:08:09.364336 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.364258 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv" Nov 03 22:08:09.374753 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.374737 5931 scope.go:122] "RemoveContainer" containerID="8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46" Nov 03 22:08:09.375031 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:08:09.375009 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46\": container with ID starting with 8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46 not found: ID does not exist" containerID="8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46" Nov 03 22:08:09.375117 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.375037 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46"} err="failed to get container status \"8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46\": rpc error: code = NotFound desc = could not find container \"8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46\": container with ID starting with 8e2e10345f9f3d8f0dfee8e8a90d6043e04f7249e86e373aa921ddd360e45b46 not found: ID does not exist" Nov 03 22:08:09.375117 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.375054 5931 scope.go:122] "RemoveContainer" containerID="65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e" Nov 03 22:08:09.384308 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.384293 5931 scope.go:122] "RemoveContainer" containerID="65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e" Nov 03 22:08:09.384517 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:08:09.384498 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e\": container with ID starting with 65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e not found: ID does not exist" containerID="65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e" Nov 03 22:08:09.384576 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.384524 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e"} err="failed to get container status \"65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e\": rpc error: code = NotFound desc = could not find container \"65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e\": container with ID starting with 65ec3f8e699e9194f4a67eb47f161c8102c60bcbf972c94367223bfd2c28558e not found: ID does not exist" Nov 03 22:08:09.451955 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451904 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.451955 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451929 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.451955 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451940 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.451955 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451948 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1419cffe-7409-4586-85b7-a40ab774dffd-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.452154 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451957 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1419cffe-7409-4586-85b7-a40ab774dffd-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.452154 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451972 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtm25\" (UniqueName: \"kubernetes.io/projected/1419cffe-7409-4586-85b7-a40ab774dffd-kube-api-access-qtm25\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.452154 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451984 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfs6\" (UniqueName: \"kubernetes.io/projected/68b6e331-b678-49ae-bd5f-9c2dba97ece7-kube-api-access-tzfs6\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.452154 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.451993 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6e331-b678-49ae-bd5f-9c2dba97ece7-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.566704 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.566671 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv"] Nov 03 22:08:09.568719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.568689 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rq7vv"] Nov 03 22:08:09.673466 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.673440 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg"] Nov 03 22:08:09.676855 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:09.676836 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-zt9vg"] Nov 03 22:08:10.505380 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:10.505351 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" path="/var/lib/kubelet/pods/1419cffe-7409-4586-85b7-a40ab774dffd/volumes" Nov 03 22:08:10.505866 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:10.505851 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" path="/var/lib/kubelet/pods/68b6e331-b678-49ae-bd5f-9c2dba97ece7/volumes" Nov 03 22:08:13.366386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:13.366298 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:15.259866 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.259837 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s"] Nov 03 22:08:15.260416 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.260402 5931 cpu_manager.go:414] "containerMap: removing container" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" Nov 03 22:08:15.260482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.260417 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" Nov 03 22:08:15.260482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.260436 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" Nov 03 22:08:15.260482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.260441 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" Nov 03 22:08:15.260642 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.260529 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b6e331-b678-49ae-bd5f-9c2dba97ece7" containerName="istio-proxy" Nov 03 22:08:15.260642 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.260540 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1419cffe-7409-4586-85b7-a40ab774dffd" containerName="istio-proxy" Nov 03 22:08:15.265495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.265475 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.302970 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.302944 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-dshm\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.303076 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.302982 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-home\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.303076 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.303062 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tls-certs\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.303211 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.303189 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72vd\" (UniqueName: \"kubernetes.io/projected/3f308068-3c70-4e1b-8fe9-04d2674b9ded-kube-api-access-m72vd\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.303265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.303249 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-model-cache\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.303337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.303327 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tmp-dir\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.317343 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.317310 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s"] Nov 03 22:08:15.363327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.363306 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-a49a32\"/\"openshift-service-ca.crt\"" Nov 03 22:08:15.363451 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.363427 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-a49a32\"/\"auth-test-model-a49a32-kserve-self-signed-certs\"" Nov 03 22:08:15.366546 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.366529 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-a49a32\"/\"kube-root-ca.crt\"" Nov 03 22:08:15.366765 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.366745 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-a49a32\"/\"default-dockercfg-fv4bs\"" Nov 03 22:08:15.404664 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.404634 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72vd\" (UniqueName: \"kubernetes.io/projected/3f308068-3c70-4e1b-8fe9-04d2674b9ded-kube-api-access-m72vd\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.404847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.404677 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-model-cache\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.404847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.404784 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tmp-dir\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.404963 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.404852 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-dshm\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.404963 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.404894 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-home\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.404963 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.404945 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tls-certs\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.405154 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.405090 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-model-cache\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.405215 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.405171 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tmp-dir\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.405269 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.405247 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-home\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.407103 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.407083 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-dshm\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.407412 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.407394 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tls-certs\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.429870 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.429843 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:15.461032 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.461005 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72vd\" (UniqueName: \"kubernetes.io/projected/3f308068-3c70-4e1b-8fe9-04d2674b9ded-kube-api-access-m72vd\") pod \"auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.579523 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.579461 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:15.959479 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:15.959452 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s"] Nov 03 22:08:15.961034 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:08:15.961010 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f308068_3c70_4e1b_8fe9_04d2674b9ded.slice/crio-ed2ab3144617bb2f6938ba063d6595bfdc0d5553772bb7ef08a4f53c1b780e96 WatchSource:0}: Error finding container ed2ab3144617bb2f6938ba063d6595bfdc0d5553772bb7ef08a4f53c1b780e96: Status 404 returned error can't find the container with id ed2ab3144617bb2f6938ba063d6595bfdc0d5553772bb7ef08a4f53c1b780e96 Nov 03 22:08:16.413784 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:16.413744 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" event={"ID":"3f308068-3c70-4e1b-8fe9-04d2674b9ded","Type":"ContainerStarted","Data":"8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca"} Nov 03 22:08:16.414198 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:16.413796 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" event={"ID":"3f308068-3c70-4e1b-8fe9-04d2674b9ded","Type":"ContainerStarted","Data":"ed2ab3144617bb2f6938ba063d6595bfdc0d5553772bb7ef08a4f53c1b780e96"} Nov 03 22:08:16.528224 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:16.528181 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" podStartSLOduration=1.528168073 podStartE2EDuration="1.528168073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:16.525070504 +0000 UTC m=+3164.135918319" watchObservedRunningTime="2026-11-03 22:08:16.528168073 +0000 UTC m=+3164.139015887" Nov 03 22:08:16.580115 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:16.580091 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:18.036601 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:18.036570 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:22.779326 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:22.779291 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:25.580596 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:25.580567 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:25.593391 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:25.593363 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:26.492423 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:26.492396 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:08:27.435019 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:27.434976 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:32.179227 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:32.179195 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:33.068232 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:08:33.068196 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:08:36.747720 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:36.747687 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:36.983103 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:36.983070 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:39.370809 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:39.370779 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:39.581337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:39.581301 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:40.364809 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:40.364776 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:42.137221 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:42.137191 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:46.858235 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:46.858204 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:51.461415 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:51.461315 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:54.139913 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:54.139881 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:56.583200 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:08:56.583164 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:01.563908 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:01.563867 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:03.067715 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:09:03.067682 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:09:06.313766 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:06.313736 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:11.149843 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:11.149809 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:15.947550 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:15.947493 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:21.035205 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:21.035166 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:22.994185 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:22.994146 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:25.736200 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:25.736165 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:27.490736 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:27.490700 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd"] Nov 03 22:09:27.841144 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:27.841117 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:28.061525 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:28.061493 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:28.321167 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:28.321099 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:30.396626 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:30.396583 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:31.602219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:31.602188 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:31.825594 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:31.825566 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:32.491508 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.491472 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" podUID="69f3a47b-f391-4ba3-ae39-c5add2a38fe0" containerName="payload-pre-processing" containerID="cri-o://8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6" gracePeriod=25 Nov 03 22:09:32.755813 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.755741 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:09:32.878585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.878557 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-plugins-config-volume\") pod \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " Nov 03 22:09:32.878816 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.878791 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-kube-api-access-86fqv\" (UniqueName: \"kubernetes.io/projected/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-kube-api-access-86fqv\") pod \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\" (UID: \"69f3a47b-f391-4ba3-ae39-c5add2a38fe0\") " Nov 03 22:09:32.878956 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.878873 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-plugins-config-volume" pod "69f3a47b-f391-4ba3-ae39-c5add2a38fe0" (UID: "69f3a47b-f391-4ba3-ae39-c5add2a38fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:32.879330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.879307 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:09:32.880920 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.880898 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-kube-api-access-86fqv" pod "69f3a47b-f391-4ba3-ae39-c5add2a38fe0" (UID: "69f3a47b-f391-4ba3-ae39-c5add2a38fe0"). InnerVolumeSpecName "kube-api-access-86fqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:32.908067 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.908041 5931 generic.go:356] "Generic (PLEG): container finished" podID="69f3a47b-f391-4ba3-ae39-c5add2a38fe0" containerID="8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6" exitCode=0 Nov 03 22:09:32.908174 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.908099 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" Nov 03 22:09:32.908174 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.908122 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" event={"ID":"69f3a47b-f391-4ba3-ae39-c5add2a38fe0","Type":"ContainerDied","Data":"8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6"} Nov 03 22:09:32.908256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.908176 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd" event={"ID":"69f3a47b-f391-4ba3-ae39-c5add2a38fe0","Type":"ContainerDied","Data":"0ff499e105239881c8cfcf80a78784c9a4b9874e2cab97397973296dc229117a"} Nov 03 22:09:32.908256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.908202 5931 scope.go:122] "RemoveContainer" containerID="8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6" Nov 03 22:09:32.919199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.919182 5931 scope.go:122] "RemoveContainer" containerID="8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6" Nov 03 22:09:32.919474 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:09:32.919455 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6\": container with ID starting with 8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6 not found: ID does not exist" containerID="8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6" Nov 03 22:09:32.919517 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.919482 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6"} err="failed to get container status \"8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6\": rpc error: code = NotFound desc = could not find container \"8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6\": container with ID starting with 8b43f5f9f434f328d82454df7b88aa857d5b98612c31f6d7d2a0eace01437ad6 not found: ID does not exist" Nov 03 22:09:32.980240 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:32.980218 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fqv\" (UniqueName: \"kubernetes.io/projected/69f3a47b-f391-4ba3-ae39-c5add2a38fe0-kube-api-access-86fqv\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:09:33.068596 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:09:33.068519 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:09:33.070154 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:33.070133 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd"] Nov 03 22:09:33.075554 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:33.075534 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-27a3c3-cbd8fbc75-s2lpd"] Nov 03 22:09:34.506299 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:34.506270 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f3a47b-f391-4ba3-ae39-c5add2a38fe0" path="/var/lib/kubelet/pods/69f3a47b-f391-4ba3-ae39-c5add2a38fe0/volumes" Nov 03 22:09:35.183628 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:35.183590 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:38.775651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:38.775597 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:39.898971 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:39.898915 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:40.330464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:40.330341 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:41.078746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:41.078719 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:41.300014 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:41.299986 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:44.561235 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:44.561187 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:49.130373 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:49.130338 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:53.867836 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:53.867794 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:54.529816 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:54.529787 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:58.548123 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:09:58.548086 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:02.592873 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:02.592838 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:03.068240 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:10:03.068160 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:10:03.351330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:03.351294 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:08.033338 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:08.033302 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:09.396718 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:09.396685 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:12.721229 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:12.721197 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:16.787732 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:16.787701 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:18.282631 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:18.282579 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:18.616021 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:18.615987 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:18.861281 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:18.861250 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:19.311761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:19.311736 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:23.721898 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:23.721859 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:25.346284 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:25.346246 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:25.589095 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:25.589052 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:25.841249 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:25.841214 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:26.337370 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:26.337272 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:27.023363 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:27.023294 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:27.244819 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:27.244786 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:28.481642 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:28.481595 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp"] Nov 03 22:10:28.482060 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:28.481869 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" containerID="cri-o://57bde4386dd45ef1ff495a817abf74f8ae2d3b6fcb6cc9c0f44b1661cf5ddf2e" gracePeriod=30 Nov 03 22:10:28.692258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:28.692226 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:28.921415 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:28.921380 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:29.258434 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:29.258367 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:32.811534 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:32.811492 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:10:32.812049 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:32.811556 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:10:33.069166 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:10:33.069079 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:10:33.641626 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.641585 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:33.754234 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.754181 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:10:33.801042 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801018 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/179b9c69-3c06-4eca-add5-47001fb043d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179b9c69-3c06-4eca-add5-47001fb043d4-istio-podinfo\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801152 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801087 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-socket\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801152 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801126 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-credential-socket\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801227 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801179 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-istio-token\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-istio-token\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801227 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801206 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-kube-api-access-pltq7\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-kube-api-access-pltq7\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801318 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801277 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-data\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801374 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801355 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-certs\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801379 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-socket" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:33.801495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801430 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-envoy\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801548 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801495 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179b9c69-3c06-4eca-add5-47001fb043d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179b9c69-3c06-4eca-add5-47001fb043d4-istiod-ca-cert\") pod \"179b9c69-3c06-4eca-add5-47001fb043d4\" (UID: \"179b9c69-3c06-4eca-add5-47001fb043d4\") " Nov 03 22:10:33.801548 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801495 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-data" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:33.801548 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801511 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-credential-socket" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:33.801689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801650 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-certs" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:33.801876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.801858 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179b9c69-3c06-4eca-add5-47001fb043d4-istiod-ca-cert" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:10:33.802191 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.802157 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.802191 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.802188 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.802330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.802204 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.802330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.802218 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.802330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.802230 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179b9c69-3c06-4eca-add5-47001fb043d4-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.803472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.803444 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/179b9c69-3c06-4eca-add5-47001fb043d4-istio-podinfo" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:10:33.803580 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.803549 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-envoy" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:33.803939 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.803920 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-kube-api-access-pltq7" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "kube-api-access-pltq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:33.803992 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.803950 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-istio-token" pod "179b9c69-3c06-4eca-add5-47001fb043d4" (UID: "179b9c69-3c06-4eca-add5-47001fb043d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:33.903072 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.903051 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.903355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.903074 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pltq7\" (UniqueName: \"kubernetes.io/projected/179b9c69-3c06-4eca-add5-47001fb043d4-kube-api-access-pltq7\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.903355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.903084 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179b9c69-3c06-4eca-add5-47001fb043d4-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:33.903355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:33.903094 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179b9c69-3c06-4eca-add5-47001fb043d4-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:10:34.069213 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:34.069150 5931 scope.go:122] "RemoveContainer" containerID="57bde4386dd45ef1ff495a817abf74f8ae2d3b6fcb6cc9c0f44b1661cf5ddf2e" Nov 03 22:10:34.322710 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:34.322636 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" event={"ID":"179b9c69-3c06-4eca-add5-47001fb043d4","Type":"ContainerDied","Data":"57bde4386dd45ef1ff495a817abf74f8ae2d3b6fcb6cc9c0f44b1661cf5ddf2e"} Nov 03 22:10:34.322710 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:34.322659 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" Nov 03 22:10:34.322875 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:34.322724 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp" event={"ID":"179b9c69-3c06-4eca-add5-47001fb043d4","Type":"ContainerDied","Data":"5b9a1fa91c9c67f2217d390a15555da68ee23a1ba20de5533ce7ba7c43f66eff"} Nov 03 22:10:34.551195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:34.551169 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp"] Nov 03 22:10:34.553797 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:34.553779 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-dfrmp"] Nov 03 22:10:35.474440 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:35.474406 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:35.784467 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:35.784380 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:10:35.788596 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:35.788575 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:10:36.506227 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:36.506197 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" path="/var/lib/kubelet/pods/179b9c69-3c06-4eca-add5-47001fb043d4/volumes" Nov 03 22:10:38.396332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:38.396296 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:43.114227 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:43.114191 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:47.836388 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:47.836353 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:49.024882 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:49.024852 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6"] Nov 03 22:10:49.163289 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:49.163252 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:49.782258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:49.782228 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:50.423693 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:50.423663 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:50.920408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:50.920370 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:52.680986 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:52.680948 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:54.375088 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:54.375054 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:54.616363 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:54.616337 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:54.861911 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:54.861885 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:57.618352 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.618318 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:57.850745 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.850711 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:57.946421 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.946348 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb"] Nov 03 22:10:57.946966 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.946951 5931 cpu_manager.go:414] "containerMap: removing container" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" Nov 03 22:10:57.947295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.946967 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" Nov 03 22:10:57.947295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.946980 5931 cpu_manager.go:414] "containerMap: removing container" podUID="69f3a47b-f391-4ba3-ae39-c5add2a38fe0" containerName="payload-pre-processing" Nov 03 22:10:57.947295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.946985 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f3a47b-f391-4ba3-ae39-c5add2a38fe0" containerName="payload-pre-processing" Nov 03 22:10:57.947295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.947072 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f3a47b-f391-4ba3-ae39-c5add2a38fe0" containerName="payload-pre-processing" Nov 03 22:10:57.947295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.947092 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="179b9c69-3c06-4eca-add5-47001fb043d4" containerName="istio-proxy" Nov 03 22:10:57.952023 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:57.952006 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.006749 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.006720 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb"] Nov 03 22:10:58.044306 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.043385 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.044306 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.043492 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkv7j\" (UniqueName: \"kubernetes.io/projected/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-kube-api-access-xkv7j\") pod \"payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.053027 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.052827 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-12dee4\"" Nov 03 22:10:58.053027 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.052993 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-12dee4-dockercfg-lpg2k\"" Nov 03 22:10:58.111944 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.111909 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7"] Nov 03 22:10:58.117342 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.117314 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.144381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.144350 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkv7j\" (UniqueName: \"kubernetes.io/projected/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-kube-api-access-xkv7j\") pod \"payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.144580 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.144536 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.145171 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.145151 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.152676 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.152595 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7"] Nov 03 22:10:58.203496 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.203431 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkv7j\" (UniqueName: \"kubernetes.io/projected/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-kube-api-access-xkv7j\") pod \"payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.245357 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.245324 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82x2b\" (UniqueName: \"kubernetes.io/projected/5e5428f9-9a24-458a-80a1-1c23be765909-kube-api-access-82x2b\") pod \"payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.245538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.245520 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5428f9-9a24-458a-80a1-1c23be765909-plugins-config-volume\") pod \"payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.266063 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.266038 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:58.349660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.349373 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5428f9-9a24-458a-80a1-1c23be765909-plugins-config-volume\") pod \"payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.349660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.349509 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x2b\" (UniqueName: \"kubernetes.io/projected/5e5428f9-9a24-458a-80a1-1c23be765909-kube-api-access-82x2b\") pod \"payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.350394 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.350368 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5428f9-9a24-458a-80a1-1c23be765909-plugins-config-volume\") pod \"payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.406009 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.405980 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x2b\" (UniqueName: \"kubernetes.io/projected/5e5428f9-9a24-458a-80a1-1c23be765909-kube-api-access-82x2b\") pod \"payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.431870 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.431838 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:58.646425 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.646399 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb"] Nov 03 22:10:58.647978 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:10:58.647934 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7a8c70_8f5c_46fc_bd8d_e91cd9f59bb6.slice/crio-8410e6bf1840426b3152ecd5e2d309f0c9eb4134597d8c241ef16abf41f33e61 WatchSource:0}: Error finding container 8410e6bf1840426b3152ecd5e2d309f0c9eb4134597d8c241ef16abf41f33e61: Status 404 returned error can't find the container with id 8410e6bf1840426b3152ecd5e2d309f0c9eb4134597d8c241ef16abf41f33e61 Nov 03 22:10:58.813304 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:58.813280 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7"] Nov 03 22:10:58.815297 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:10:58.815263 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5428f9_9a24_458a_80a1_1c23be765909.slice/crio-4926e800960e6c74fc37d597f94642a4c98453cb2b43c7d486df4af01fe1b118 WatchSource:0}: Error finding container 4926e800960e6c74fc37d597f94642a4c98453cb2b43c7d486df4af01fe1b118: Status 404 returned error can't find the container with id 4926e800960e6c74fc37d597f94642a4c98453cb2b43c7d486df4af01fe1b118 Nov 03 22:10:59.498578 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.498546 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" event={"ID":"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6","Type":"ContainerStarted","Data":"cc0efffad83b007d3c128ef699591935f39ae213a68e8650176c5193ecf4bb2e"} Nov 03 22:10:59.498803 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.498597 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" event={"ID":"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6","Type":"ContainerStarted","Data":"8410e6bf1840426b3152ecd5e2d309f0c9eb4134597d8c241ef16abf41f33e61"} Nov 03 22:10:59.498803 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.498773 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:10:59.500545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.500516 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" event={"ID":"5e5428f9-9a24-458a-80a1-1c23be765909","Type":"ContainerStarted","Data":"e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29"} Nov 03 22:10:59.500713 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.500571 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" event={"ID":"5e5428f9-9a24-458a-80a1-1c23be765909","Type":"ContainerStarted","Data":"4926e800960e6c74fc37d597f94642a4c98453cb2b43c7d486df4af01fe1b118"} Nov 03 22:10:59.500713 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.500600 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:10:59.612644 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.612551 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" podStartSLOduration=2.61253846 podStartE2EDuration="2.61253846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:59.610659175 +0000 UTC m=+3327.221506992" watchObservedRunningTime="2026-11-03 22:10:59.61253846 +0000 UTC m=+3327.223386273" Nov 03 22:10:59.726796 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:10:59.726736 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" podStartSLOduration=1.726718899 podStartE2EDuration="1.726718899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:59.723523404 +0000 UTC m=+3327.334371221" watchObservedRunningTime="2026-11-03 22:10:59.726718899 +0000 UTC m=+3327.337566713" Nov 03 22:11:00.635895 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:00.635866 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:02.663407 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:02.663372 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:03.068299 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:11:03.068223 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:11:04.045820 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.045786 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" podUID="3823ed9d-12a2-4b8f-81cf-691926b28a43" containerName="main" containerID="cri-o://6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308" gracePeriod=45 Nov 03 22:11:04.298245 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.298192 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:11:04.323517 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323494 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-home\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-home\") pod \"3823ed9d-12a2-4b8f-81cf-691926b28a43\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " Nov 03 22:11:04.323663 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323653 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3823ed9d-12a2-4b8f-81cf-691926b28a43-tls-certs\" (UniqueName: \"kubernetes.io/secret/3823ed9d-12a2-4b8f-81cf-691926b28a43-tls-certs\") pod \"3823ed9d-12a2-4b8f-81cf-691926b28a43\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " Nov 03 22:11:04.323730 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323684 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-tmp-dir\") pod \"3823ed9d-12a2-4b8f-81cf-691926b28a43\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " Nov 03 22:11:04.323730 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323711 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3823ed9d-12a2-4b8f-81cf-691926b28a43-kube-api-access-xh6v6\" (UniqueName: \"kubernetes.io/projected/3823ed9d-12a2-4b8f-81cf-691926b28a43-kube-api-access-xh6v6\") pod \"3823ed9d-12a2-4b8f-81cf-691926b28a43\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " Nov 03 22:11:04.323845 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323732 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-model-cache\") pod \"3823ed9d-12a2-4b8f-81cf-691926b28a43\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " Nov 03 22:11:04.323845 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323784 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-dshm\") pod \"3823ed9d-12a2-4b8f-81cf-691926b28a43\" (UID: \"3823ed9d-12a2-4b8f-81cf-691926b28a43\") " Nov 03 22:11:04.323950 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323849 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-home" pod "3823ed9d-12a2-4b8f-81cf-691926b28a43" (UID: "3823ed9d-12a2-4b8f-81cf-691926b28a43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:04.324000 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323956 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-model-cache" pod "3823ed9d-12a2-4b8f-81cf-691926b28a43" (UID: "3823ed9d-12a2-4b8f-81cf-691926b28a43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:04.324000 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.323968 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-tmp-dir" pod "3823ed9d-12a2-4b8f-81cf-691926b28a43" (UID: "3823ed9d-12a2-4b8f-81cf-691926b28a43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:04.324393 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.324332 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:11:04.324393 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.324356 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:11:04.324393 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.324370 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:11:04.325927 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.325899 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3823ed9d-12a2-4b8f-81cf-691926b28a43-tls-certs" pod "3823ed9d-12a2-4b8f-81cf-691926b28a43" (UID: "3823ed9d-12a2-4b8f-81cf-691926b28a43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:11:04.326301 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.326278 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-dshm" pod "3823ed9d-12a2-4b8f-81cf-691926b28a43" (UID: "3823ed9d-12a2-4b8f-81cf-691926b28a43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:04.326382 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.326336 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3823ed9d-12a2-4b8f-81cf-691926b28a43-kube-api-access-xh6v6" pod "3823ed9d-12a2-4b8f-81cf-691926b28a43" (UID: "3823ed9d-12a2-4b8f-81cf-691926b28a43"). InnerVolumeSpecName "kube-api-access-xh6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:11:04.426213 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.426186 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3823ed9d-12a2-4b8f-81cf-691926b28a43-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:11:04.426336 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.426218 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6v6\" (UniqueName: \"kubernetes.io/projected/3823ed9d-12a2-4b8f-81cf-691926b28a43-kube-api-access-xh6v6\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:11:04.426336 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.426234 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3823ed9d-12a2-4b8f-81cf-691926b28a43-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:11:04.533386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.533352 5931 generic.go:356] "Generic (PLEG): container finished" podID="3823ed9d-12a2-4b8f-81cf-691926b28a43" containerID="6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308" exitCode=0 Nov 03 22:11:04.533482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.533395 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" Nov 03 22:11:04.533482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.533442 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" event={"ID":"3823ed9d-12a2-4b8f-81cf-691926b28a43","Type":"ContainerDied","Data":"6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308"} Nov 03 22:11:04.533570 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.533481 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6" event={"ID":"3823ed9d-12a2-4b8f-81cf-691926b28a43","Type":"ContainerDied","Data":"4cba611c950d16151197624e25a0752529be15e393478f7a50ec013f9f2820a4"} Nov 03 22:11:04.533570 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.533497 5931 scope.go:122] "RemoveContainer" containerID="6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308" Nov 03 22:11:04.542973 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.542958 5931 scope.go:122] "RemoveContainer" containerID="6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308" Nov 03 22:11:04.543228 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:11:04.543209 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308\": container with ID starting with 6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308 not found: ID does not exist" containerID="6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308" Nov 03 22:11:04.543298 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.543237 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308"} err="failed to get container status \"6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308\": rpc error: code = NotFound desc = could not find container \"6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308\": container with ID starting with 6efc868b1300b9127e32906b0e7173c58d81afb452aabbb251e3ef2f07adc308 not found: ID does not exist" Nov 03 22:11:04.696735 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.696715 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6"] Nov 03 22:11:04.699381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:04.699364 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ddbba/auth-test-model-6ddbba-kserve-54b569dddf-967s6"] Nov 03 22:11:06.507063 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:06.507023 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3823ed9d-12a2-4b8f-81cf-691926b28a43" path="/var/lib/kubelet/pods/3823ed9d-12a2-4b8f-81cf-691926b28a43/volumes" Nov 03 22:11:08.326219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:08.326185 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:10.509664 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:10.509636 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:11:10.510045 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:10.509789 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:11:11.955846 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:11.955773 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:16.649759 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:16.649728 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:21.349313 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:21.349276 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:26.034160 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:26.034125 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:30.759553 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:30.759518 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:33.068689 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:11:33.068650 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:11:35.707120 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:35.707087 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:40.407328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:40.407289 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:42.234256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:42.234220 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:42.613593 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:42.613561 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:42.840833 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:42.840800 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:45.075847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:45.075811 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:47.771891 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:47.771853 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:49.784399 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:49.784361 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:54.443949 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:54.443913 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:55.195756 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:55.195726 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:59.072595 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:59.072551 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:59.296422 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:11:59.296391 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:03.074573 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:12:03.074537 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:12:03.916974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:03.916937 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:04.230794 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:04.230709 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:08.948581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:08.948540 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:10.215836 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:10.215807 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:13.638748 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:13.638705 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:18.339099 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:18.339064 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:23.119924 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:23.119892 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:24.613033 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:24.613003 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:27.668002 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:27.667963 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:28.509603 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:28.509570 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:29.141362 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:29.141311 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:29.638078 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:29.638046 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:32.491210 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:32.491181 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:33.068327 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:12:33.068293 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:12:33.371958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:33.371903 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:33.626980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:33.626915 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:37.006424 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.006395 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:37.247883 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.247832 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:37.328793 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.328721 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv"] Nov 03 22:12:37.329409 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.329390 5931 cpu_manager.go:414] "containerMap: removing container" podUID="3823ed9d-12a2-4b8f-81cf-691926b28a43" containerName="main" Nov 03 22:12:37.329511 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.329411 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3823ed9d-12a2-4b8f-81cf-691926b28a43" containerName="main" Nov 03 22:12:37.329628 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.329596 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3823ed9d-12a2-4b8f-81cf-691926b28a43" containerName="main" Nov 03 22:12:37.333134 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.333108 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.388536 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.388515 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv"] Nov 03 22:12:37.428834 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.428793 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.429165 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.428896 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddmq\" (UniqueName: \"kubernetes.io/projected/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-kube-api-access-8ddmq\") pod \"payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.434424 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.434389 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-5b1489\"" Nov 03 22:12:37.434659 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.434507 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-5b1489-dockercfg-vsfjh\"" Nov 03 22:12:37.486951 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.486919 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8"] Nov 03 22:12:37.491150 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.491130 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.530195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.530166 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5ls\" (UniqueName: \"kubernetes.io/projected/3b27ab41-5187-4091-b628-b56d17fb8658-kube-api-access-fw5ls\") pod \"payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.530333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.530267 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b27ab41-5187-4091-b628-b56d17fb8658-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.530333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.530320 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.530451 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.530405 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddmq\" (UniqueName: \"kubernetes.io/projected/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-kube-api-access-8ddmq\") pod \"payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.531360 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.531336 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.531604 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.531586 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8"] Nov 03 22:12:37.586954 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.586898 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddmq\" (UniqueName: \"kubernetes.io/projected/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-kube-api-access-8ddmq\") pod \"payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.631140 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.631108 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5ls\" (UniqueName: \"kubernetes.io/projected/3b27ab41-5187-4091-b628-b56d17fb8658-kube-api-access-fw5ls\") pod \"payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.631258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.631181 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b27ab41-5187-4091-b628-b56d17fb8658-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.631790 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.631772 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b27ab41-5187-4091-b628-b56d17fb8658-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.646839 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.646817 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:37.688681 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.688655 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5ls\" (UniqueName: \"kubernetes.io/projected/3b27ab41-5187-4091-b628-b56d17fb8658-kube-api-access-fw5ls\") pod \"payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.805171 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.805137 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:37.855793 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:37.855762 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:38.020065 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:38.020040 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv"] Nov 03 22:12:38.021733 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:12:38.021709 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f5ffad_a1ef_4381_b6e9_4d133b50ad10.slice/crio-e84e55d62d0fcc457a5d7487721c3e369be7ac98902ff0de53dab8ee761d7d70 WatchSource:0}: Error finding container e84e55d62d0fcc457a5d7487721c3e369be7ac98902ff0de53dab8ee761d7d70: Status 404 returned error can't find the container with id e84e55d62d0fcc457a5d7487721c3e369be7ac98902ff0de53dab8ee761d7d70 Nov 03 22:12:38.144490 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:38.144431 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" event={"ID":"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10","Type":"ContainerStarted","Data":"b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6"} Nov 03 22:12:38.144644 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:38.144491 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:38.144644 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:38.144513 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" event={"ID":"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10","Type":"ContainerStarted","Data":"e84e55d62d0fcc457a5d7487721c3e369be7ac98902ff0de53dab8ee761d7d70"} Nov 03 22:12:38.182893 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:38.182872 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8"] Nov 03 22:12:38.183804 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:12:38.183780 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b27ab41_5187_4091_b628_b56d17fb8658.slice/crio-891cf5949f562d99ec1d57b8d380a8da318188f781f22a58c0ce4bc88dc6ef67 WatchSource:0}: Error finding container 891cf5949f562d99ec1d57b8d380a8da318188f781f22a58c0ce4bc88dc6ef67: Status 404 returned error can't find the container with id 891cf5949f562d99ec1d57b8d380a8da318188f781f22a58c0ce4bc88dc6ef67 Nov 03 22:12:38.258110 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:38.258060 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" podStartSLOduration=1.2580438250000001 podStartE2EDuration="1.258043825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:38.255578109 +0000 UTC m=+3425.866425924" watchObservedRunningTime="2026-11-03 22:12:38.258043825 +0000 UTC m=+3425.868891640" Nov 03 22:12:39.153073 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:39.153037 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" event={"ID":"3b27ab41-5187-4091-b628-b56d17fb8658","Type":"ContainerStarted","Data":"66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30"} Nov 03 22:12:39.153480 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:39.153094 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" event={"ID":"3b27ab41-5187-4091-b628-b56d17fb8658","Type":"ContainerStarted","Data":"891cf5949f562d99ec1d57b8d380a8da318188f781f22a58c0ce4bc88dc6ef67"} Nov 03 22:12:39.153480 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:39.153293 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:39.273980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:39.273930 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" podStartSLOduration=2.273918298 podStartE2EDuration="2.273918298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:39.270989297 +0000 UTC m=+3426.881837111" watchObservedRunningTime="2026-11-03 22:12:39.273918298 +0000 UTC m=+3426.884766112" Nov 03 22:12:42.846426 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:42.846349 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:47.639903 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:47.639872 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:49.156437 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:49.156410 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:12:50.162125 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:50.162100 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:12:50.923522 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:50.923488 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx"] Nov 03 22:12:50.923727 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:50.923708 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" containerID="cri-o://cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f" gracePeriod=30 Nov 03 22:12:52.315226 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:52.315195 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:54.829048 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:54.829011 5931 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:12:54.829522 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:54.829074 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:12:55.541018 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:55.540985 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:56.189125 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.189072 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:12:56.261629 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.261585 5931 generic.go:356] "Generic (PLEG): container finished" podID="b501f81c-7184-4bef-a245-aa82d97673d8" containerID="cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f" exitCode=0 Nov 03 22:12:56.261761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.261642 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" Nov 03 22:12:56.261761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.261684 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" event={"ID":"b501f81c-7184-4bef-a245-aa82d97673d8","Type":"ContainerDied","Data":"cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f"} Nov 03 22:12:56.261761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.261736 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx" event={"ID":"b501f81c-7184-4bef-a245-aa82d97673d8","Type":"ContainerDied","Data":"bee537d49d9d9747a285d34fe120eb86f631da7448f82d9f8079ab95e30a701e"} Nov 03 22:12:56.261761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.261756 5931 scope.go:122] "RemoveContainer" containerID="cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f" Nov 03 22:12:56.271481 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.271464 5931 scope.go:122] "RemoveContainer" containerID="cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f" Nov 03 22:12:56.271787 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:12:56.271769 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f\": container with ID starting with cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f not found: ID does not exist" containerID="cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f" Nov 03 22:12:56.271864 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.271793 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f"} err="failed to get container status \"cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f\": rpc error: code = NotFound desc = could not find container \"cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f\": container with ID starting with cf54e724ece40a220c07ff6157587e64af3b70d33e2dce1db0bd1aa9005c047f not found: ID does not exist" Nov 03 22:12:56.323283 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323263 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b501f81c-7184-4bef-a245-aa82d97673d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b501f81c-7184-4bef-a245-aa82d97673d8-istio-podinfo\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323389 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323299 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b501f81c-7184-4bef-a245-aa82d97673d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b501f81c-7184-4bef-a245-aa82d97673d8-istiod-ca-cert\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323389 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323322 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-kube-api-access-nm59l\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-kube-api-access-nm59l\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323389 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323381 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-data\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323556 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323518 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-socket\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323556 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323550 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-credential-socket\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323593 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-certs\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323660 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-envoy\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323789 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323702 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-istio-token\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-istio-token\") pod \"b501f81c-7184-4bef-a245-aa82d97673d8\" (UID: \"b501f81c-7184-4bef-a245-aa82d97673d8\") " Nov 03 22:12:56.323789 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323761 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-data" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:56.323789 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.323768 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b501f81c-7184-4bef-a245-aa82d97673d8-istiod-ca-cert" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:12:56.324141 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.324047 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-certs" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:56.324225 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.324142 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b501f81c-7184-4bef-a245-aa82d97673d8-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.324300 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.324242 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.324376 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.324336 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-credential-socket" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:56.324450 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.324397 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-socket" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:56.325718 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.325684 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-kube-api-access-nm59l" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "kube-api-access-nm59l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:56.326111 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.326082 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b501f81c-7184-4bef-a245-aa82d97673d8-istio-podinfo" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:12:56.326214 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.326176 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-istio-token" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:56.326397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.326372 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-envoy" pod "b501f81c-7184-4bef-a245-aa82d97673d8" (UID: "b501f81c-7184-4bef-a245-aa82d97673d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:56.425754 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.425726 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.425876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.425758 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.425876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.425773 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.425876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.425786 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b501f81c-7184-4bef-a245-aa82d97673d8-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.425876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.425800 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.425876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.425812 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b501f81c-7184-4bef-a245-aa82d97673d8-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.425876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.425825 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm59l\" (UniqueName: \"kubernetes.io/projected/b501f81c-7184-4bef-a245-aa82d97673d8-kube-api-access-nm59l\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:12:56.675039 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.675009 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx"] Nov 03 22:12:56.681770 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:56.681743 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-rprtx"] Nov 03 22:12:57.040507 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:57.040478 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:58.387897 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:58.387868 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:58.505800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:12:58.505774 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" path="/var/lib/kubelet/pods/b501f81c-7184-4bef-a245-aa82d97673d8/volumes" Nov 03 22:13:01.732087 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:01.732052 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:03.068027 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:13:03.067979 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:13:06.383573 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:06.383544 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:11.058342 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:11.058308 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:15.737574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:15.737523 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:17.540607 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:17.540571 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:20.723585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:20.723553 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:25.453404 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:25.453367 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:26.684705 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.684673 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27"] Nov 03 22:13:26.685378 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.685359 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" Nov 03 22:13:26.685423 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.685379 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" Nov 03 22:13:26.685496 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.685487 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b501f81c-7184-4bef-a245-aa82d97673d8" containerName="istio-proxy" Nov 03 22:13:26.689036 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.689020 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.793569 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.793516 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-12dee4\"/\"kube-root-ca.crt\"" Nov 03 22:13:26.793569 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.793557 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-12dee4\"/\"openshift-service-ca.crt\"" Nov 03 22:13:26.793810 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.793644 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-12dee4\"/\"default-dockercfg-jpd4d\"" Nov 03 22:13:26.793810 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.793681 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-12dee4\"/\"autores-model-12dee4-kserve-self-signed-certs\"" Nov 03 22:13:26.796470 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.796451 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27"] Nov 03 22:13:26.826012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.825983 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-tmp-dir\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.826127 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.826023 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-dshm\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.826212 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.826194 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-home\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.826266 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.826224 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-model-cache\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.826323 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.826273 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b722\" (UniqueName: \"kubernetes.io/projected/0637122d-1b20-4e4c-9961-9d582cccf487-kube-api-access-7b722\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.826378 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.826362 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0637122d-1b20-4e4c-9961-9d582cccf487-tls-certs\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.877908 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.877877 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:26.927397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927359 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-tmp-dir\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927405 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-dshm\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927590 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927500 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-home\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927590 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927517 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-model-cache\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927590 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927558 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b722\" (UniqueName: \"kubernetes.io/projected/0637122d-1b20-4e4c-9961-9d582cccf487-kube-api-access-7b722\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927728 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927592 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0637122d-1b20-4e4c-9961-9d582cccf487-tls-certs\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927843 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927820 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-tmp-dir\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927895 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927880 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-home\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.927974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.927955 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-model-cache\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.929635 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.929600 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-dshm\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.929971 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.929955 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0637122d-1b20-4e4c-9961-9d582cccf487-tls-certs\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:26.984505 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:26.984449 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b722\" (UniqueName: \"kubernetes.io/projected/0637122d-1b20-4e4c-9961-9d582cccf487-kube-api-access-7b722\") pod \"autores-model-12dee4-kserve-6d9c864c57-4dx27\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:27.001705 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:27.001689 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:27.384907 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:27.384885 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27"] Nov 03 22:13:27.386088 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:13:27.386054 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0637122d_1b20_4e4c_9961_9d582cccf487.slice/crio-b32d39005bef20ecdd12e33b73c9af84a834a9488823f40cb1a8bda819a0e48d WatchSource:0}: Error finding container b32d39005bef20ecdd12e33b73c9af84a834a9488823f40cb1a8bda819a0e48d: Status 404 returned error can't find the container with id b32d39005bef20ecdd12e33b73c9af84a834a9488823f40cb1a8bda819a0e48d Nov 03 22:13:27.472464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:27.472433 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" event={"ID":"0637122d-1b20-4e4c-9961-9d582cccf487","Type":"ContainerStarted","Data":"68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7"} Nov 03 22:13:27.472577 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:27.472479 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" event={"ID":"0637122d-1b20-4e4c-9961-9d582cccf487","Type":"ContainerStarted","Data":"b32d39005bef20ecdd12e33b73c9af84a834a9488823f40cb1a8bda819a0e48d"} Nov 03 22:13:27.585574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:27.585434 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" podStartSLOduration=1.58541878 podStartE2EDuration="1.58541878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:27.584893657 +0000 UTC m=+3475.195741471" watchObservedRunningTime="2026-11-03 22:13:27.58541878 +0000 UTC m=+3475.196266594" Nov 03 22:13:28.001875 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:28.001844 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:30.155893 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:30.155861 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:33.069924 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:13:33.069891 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:13:34.081533 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:34.081494 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:34.304395 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:34.304366 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:34.926730 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:34.926698 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:37.001976 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:37.001949 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:37.014759 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:37.014734 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:37.548277 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:37.548252 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:13:39.754573 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:39.754543 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:39.984397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:39.984371 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:44.567096 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:44.567065 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:44.799376 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:44.799347 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:49.639402 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:49.639370 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:54.351392 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:54.351357 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:55.882311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:55.882281 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:59.038492 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:13:59.038458 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:03.068726 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:14:03.068688 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:14:03.810747 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:03.810714 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:08.782322 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:08.782286 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:13.454222 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:13.454147 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:18.122792 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:18.122756 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:22.853437 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:22.853405 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:27.654314 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:27.654279 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:32.027620 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:32.027573 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:32.536482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:32.536450 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:33.068468 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:14:33.068429 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:14:37.462912 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:37.462877 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:37.699506 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:37.699474 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:42.228707 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:42.228676 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:46.925049 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:46.925000 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:51.122001 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:51.121970 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:51.848875 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:51.848836 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:52.078682 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:52.078647 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:52.519336 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:52.519310 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:54.497980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:54.497949 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:56.141995 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:56.141963 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:56.384154 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:56.384127 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:56.877666 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:56.877635 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:59.402427 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:59.402398 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:59.631887 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:14:59.631860 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:00.162448 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.162422 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp"] Nov 03 22:15:00.166448 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.166432 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.189346 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.189319 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l"] Nov 03 22:15:00.195923 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.195900 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.204147 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.204129 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq"] Nov 03 22:15:00.208276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.208259 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.211163 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.211142 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh"] Nov 03 22:15:00.215135 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.215117 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.258413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.258385 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp"] Nov 03 22:15:00.263528 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.263501 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hg8h\" (UniqueName: \"kubernetes.io/projected/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-kube-api-access-8hg8h\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.263697 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.263569 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd644ea6-af94-46b1-935f-019350f24563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.263767 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.263736 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvjb\" (UniqueName: \"kubernetes.io/projected/87832f8b-cef0-42e9-afd0-98db0e82689f-kube-api-access-hbvjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.263855 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.263839 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.263940 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.263920 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.264002 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.263957 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87832f8b-cef0-42e9-afd0-98db0e82689f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.264091 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.264075 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcb2z\" (UniqueName: \"kubernetes.io/projected/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-kube-api-access-fcb2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.264147 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.264109 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cvl\" (UniqueName: \"kubernetes.io/projected/cd644ea6-af94-46b1-935f-019350f24563-kube-api-access-m2cvl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.271148 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.271124 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l"] Nov 03 22:15:00.275529 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.275503 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq"] Nov 03 22:15:00.279908 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.279881 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh"] Nov 03 22:15:00.319394 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.319375 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\"" Nov 03 22:15:00.365200 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365176 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hg8h\" (UniqueName: \"kubernetes.io/projected/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-kube-api-access-8hg8h\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.365316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365216 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd644ea6-af94-46b1-935f-019350f24563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.365316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365239 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvjb\" (UniqueName: \"kubernetes.io/projected/87832f8b-cef0-42e9-afd0-98db0e82689f-kube-api-access-hbvjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.365316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365278 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.365316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365303 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.365495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365321 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87832f8b-cef0-42e9-afd0-98db0e82689f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.365495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365393 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcb2z\" (UniqueName: \"kubernetes.io/projected/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-kube-api-access-fcb2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.365495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365420 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cvl\" (UniqueName: \"kubernetes.io/projected/cd644ea6-af94-46b1-935f-019350f24563-kube-api-access-m2cvl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.365954 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.365928 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd644ea6-af94-46b1-935f-019350f24563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.366044 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.366002 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.366124 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.366106 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.366285 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.366269 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87832f8b-cef0-42e9-afd0-98db0e82689f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.423295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.423238 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcb2z\" (UniqueName: \"kubernetes.io/projected/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-kube-api-access-fcb2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.423715 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.423521 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cvl\" (UniqueName: \"kubernetes.io/projected/cd644ea6-af94-46b1-935f-019350f24563-kube-api-access-m2cvl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.423715 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.423648 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hg8h\" (UniqueName: \"kubernetes.io/projected/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-kube-api-access-8hg8h\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.425198 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.425176 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvjb\" (UniqueName: \"kubernetes.io/projected/87832f8b-cef0-42e9-afd0-98db0e82689f-kube-api-access-hbvjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.480277 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.480255 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:00.510352 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.510330 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:00.521454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.521435 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:00.532685 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.532663 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:00.883338 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.876832 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp"] Nov 03 22:15:00.883338 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:15:00.878700 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ee43eb_927d_4a38_89f4_fcd2c0ca7087.slice/crio-0a931d4e2d61d8a8aaba0a5c63bb1c3803890f8d99527d36f0c976e137934d8f WatchSource:0}: Error finding container 0a931d4e2d61d8a8aaba0a5c63bb1c3803890f8d99527d36f0c976e137934d8f: Status 404 returned error can't find the container with id 0a931d4e2d61d8a8aaba0a5c63bb1c3803890f8d99527d36f0c976e137934d8f Nov 03 22:15:00.926623 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.926590 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l"] Nov 03 22:15:00.928426 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:15:00.928404 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af1c38c_cdf7_4c99_ab1c_dae3d5ba7b75.slice/crio-b75f7915590e255e367c4a48a990536d693e2789e1ef99f3697dce5b5707fff2 WatchSource:0}: Error finding container b75f7915590e255e367c4a48a990536d693e2789e1ef99f3697dce5b5707fff2: Status 404 returned error can't find the container with id b75f7915590e255e367c4a48a990536d693e2789e1ef99f3697dce5b5707fff2 Nov 03 22:15:00.959960 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.959862 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh"] Nov 03 22:15:00.961413 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:15:00.961390 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd644ea6_af94_46b1_935f_019350f24563.slice/crio-5e91b83a3997d507ab3659405bc2cdf3949df6dbb8e783bce0fd37676492311e WatchSource:0}: Error finding container 5e91b83a3997d507ab3659405bc2cdf3949df6dbb8e783bce0fd37676492311e: Status 404 returned error can't find the container with id 5e91b83a3997d507ab3659405bc2cdf3949df6dbb8e783bce0fd37676492311e Nov 03 22:15:00.980073 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:15:00.980052 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87832f8b_cef0_42e9_afd0_98db0e82689f.slice/crio-131109c6b1157bb1851a23c5eaf83f9b5ac0cad981071835d3d45abede37643e WatchSource:0}: Error finding container 131109c6b1157bb1851a23c5eaf83f9b5ac0cad981071835d3d45abede37643e: Status 404 returned error can't find the container with id 131109c6b1157bb1851a23c5eaf83f9b5ac0cad981071835d3d45abede37643e Nov 03 22:15:00.980167 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:00.980105 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq"] Nov 03 22:15:01.077131 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:01.077108 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" event={"ID":"cd644ea6-af94-46b1-935f-019350f24563","Type":"ContainerStarted","Data":"5e91b83a3997d507ab3659405bc2cdf3949df6dbb8e783bce0fd37676492311e"} Nov 03 22:15:01.078376 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:01.078351 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" event={"ID":"50ee43eb-927d-4a38-89f4-fcd2c0ca7087","Type":"ContainerStarted","Data":"0a931d4e2d61d8a8aaba0a5c63bb1c3803890f8d99527d36f0c976e137934d8f"} Nov 03 22:15:01.079709 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:01.079684 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" event={"ID":"87832f8b-cef0-42e9-afd0-98db0e82689f","Type":"ContainerStarted","Data":"131109c6b1157bb1851a23c5eaf83f9b5ac0cad981071835d3d45abede37643e"} Nov 03 22:15:01.081192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:01.081169 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" event={"ID":"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75","Type":"ContainerStarted","Data":"b75f7915590e255e367c4a48a990536d693e2789e1ef99f3697dce5b5707fff2"} Nov 03 22:15:01.756242 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:01.756156 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s"] Nov 03 22:15:01.866497 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:01.866461 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:02.092538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.092514 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:02.094008 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.093987 5931 generic.go:356] "Generic (PLEG): container finished" podID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerID="fb117c9483af39ff1bf61209f1f48a612a47d0efaf38b4ec3ba89d3dc1fc1d54" exitCode=60 Nov 03 22:15:02.094119 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.094035 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" event={"ID":"87832f8b-cef0-42e9-afd0-98db0e82689f","Type":"ContainerDied","Data":"fb117c9483af39ff1bf61209f1f48a612a47d0efaf38b4ec3ba89d3dc1fc1d54"} Nov 03 22:15:02.094307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.094290 5931 scope.go:122] "RemoveContainer" containerID="fb117c9483af39ff1bf61209f1f48a612a47d0efaf38b4ec3ba89d3dc1fc1d54" Nov 03 22:15:02.095778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.095754 5931 generic.go:356] "Generic (PLEG): container finished" podID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerID="e0af74504fcb380d57a459e88e7f1536f67f9271cbbb84d91eceaa22125095b3" exitCode=60 Nov 03 22:15:02.095854 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.095836 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" event={"ID":"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75","Type":"ContainerDied","Data":"e0af74504fcb380d57a459e88e7f1536f67f9271cbbb84d91eceaa22125095b3"} Nov 03 22:15:02.096027 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.096002 5931 scope.go:122] "RemoveContainer" containerID="e0af74504fcb380d57a459e88e7f1536f67f9271cbbb84d91eceaa22125095b3" Nov 03 22:15:02.097628 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.097582 5931 generic.go:356] "Generic (PLEG): container finished" podID="cd644ea6-af94-46b1-935f-019350f24563" containerID="196f713d487f173bfcb6d5c2a8ff1006614b006cdbdaef62b1cb62e9a6b4d6ae" exitCode=60 Nov 03 22:15:02.097729 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.097690 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" event={"ID":"cd644ea6-af94-46b1-935f-019350f24563","Type":"ContainerDied","Data":"196f713d487f173bfcb6d5c2a8ff1006614b006cdbdaef62b1cb62e9a6b4d6ae"} Nov 03 22:15:02.097881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.097868 5931 scope.go:122] "RemoveContainer" containerID="196f713d487f173bfcb6d5c2a8ff1006614b006cdbdaef62b1cb62e9a6b4d6ae" Nov 03 22:15:02.103406 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.103385 5931 generic.go:356] "Generic (PLEG): container finished" podID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerID="17d82362fb3e687b6d92fca638937b8bffb218a1c95c69207e91975fcbd80e41" exitCode=60 Nov 03 22:15:02.103500 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.103434 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" event={"ID":"50ee43eb-927d-4a38-89f4-fcd2c0ca7087","Type":"ContainerDied","Data":"17d82362fb3e687b6d92fca638937b8bffb218a1c95c69207e91975fcbd80e41"} Nov 03 22:15:02.103709 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:02.103694 5931 scope.go:122] "RemoveContainer" containerID="17d82362fb3e687b6d92fca638937b8bffb218a1c95c69207e91975fcbd80e41" Nov 03 22:15:03.069103 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:03.069026 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:15:03.113574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.113545 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" event={"ID":"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75","Type":"ContainerStarted","Data":"37259d4b9731eae4e7c1f98867c27e127270c13f463161134782194b92101685"} Nov 03 22:15:03.115700 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.115675 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" event={"ID":"cd644ea6-af94-46b1-935f-019350f24563","Type":"ContainerStarted","Data":"41df8a82970d4edc87694e09b230adffb6cb39304cc70ab973616ca8500fbd1e"} Nov 03 22:15:03.117581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.117556 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" event={"ID":"50ee43eb-927d-4a38-89f4-fcd2c0ca7087","Type":"ContainerStarted","Data":"0d4c2fe710694ada68ba66c59365fdc2ede11ae0c058ed3e44ebc55c5afca8ce"} Nov 03 22:15:03.119432 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.119408 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" event={"ID":"87832f8b-cef0-42e9-afd0-98db0e82689f","Type":"ContainerStarted","Data":"24f8967ddc1e109405ba19266ac90fbd403cddd5d0c81987923fd0f97735e3ca"} Nov 03 22:15:03.225374 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.225327 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" podStartSLOduration=3.225309626 podStartE2EDuration="3.225309626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:03.224041073 +0000 UTC m=+3570.834888884" watchObservedRunningTime="2026-11-03 22:15:03.225309626 +0000 UTC m=+3570.836157442" Nov 03 22:15:03.337455 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.337415 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" podStartSLOduration=3.337403247 podStartE2EDuration="3.337403247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:03.335105279 +0000 UTC m=+3570.945953093" watchObservedRunningTime="2026-11-03 22:15:03.337403247 +0000 UTC m=+3570.948251062" Nov 03 22:15:03.449639 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.449579 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" podStartSLOduration=3.449566431 podStartE2EDuration="3.449566431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:03.44764478 +0000 UTC m=+3571.058492594" watchObservedRunningTime="2026-11-03 22:15:03.449566431 +0000 UTC m=+3571.060414294" Nov 03 22:15:03.560485 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:03.560442 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" podStartSLOduration=3.560429574 podStartE2EDuration="3.560429574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:03.558051929 +0000 UTC m=+3571.168899744" watchObservedRunningTime="2026-11-03 22:15:03.560429574 +0000 UTC m=+3571.171277386" Nov 03 22:15:04.128946 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.128909 5931 generic.go:356] "Generic (PLEG): container finished" podID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerID="24f8967ddc1e109405ba19266ac90fbd403cddd5d0c81987923fd0f97735e3ca" exitCode=60 Nov 03 22:15:04.129400 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.128956 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" event={"ID":"87832f8b-cef0-42e9-afd0-98db0e82689f","Type":"ContainerDied","Data":"24f8967ddc1e109405ba19266ac90fbd403cddd5d0c81987923fd0f97735e3ca"} Nov 03 22:15:04.129400 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.129038 5931 scope.go:122] "RemoveContainer" containerID="fb117c9483af39ff1bf61209f1f48a612a47d0efaf38b4ec3ba89d3dc1fc1d54" Nov 03 22:15:04.129400 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.129321 5931 scope.go:122] "RemoveContainer" containerID="24f8967ddc1e109405ba19266ac90fbd403cddd5d0c81987923fd0f97735e3ca" Nov 03 22:15:04.129629 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:04.129593 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq_odh-ai-gateway-infra(87832f8b-cef0-42e9-afd0-98db0e82689f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" Nov 03 22:15:04.131346 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.131322 5931 generic.go:356] "Generic (PLEG): container finished" podID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerID="37259d4b9731eae4e7c1f98867c27e127270c13f463161134782194b92101685" exitCode=60 Nov 03 22:15:04.131450 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.131399 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" event={"ID":"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75","Type":"ContainerDied","Data":"37259d4b9731eae4e7c1f98867c27e127270c13f463161134782194b92101685"} Nov 03 22:15:04.131690 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.131673 5931 scope.go:122] "RemoveContainer" containerID="37259d4b9731eae4e7c1f98867c27e127270c13f463161134782194b92101685" Nov 03 22:15:04.131924 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:04.131901 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l_odh-ai-gateway-infra(7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" Nov 03 22:15:04.133435 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.133406 5931 generic.go:356] "Generic (PLEG): container finished" podID="cd644ea6-af94-46b1-935f-019350f24563" containerID="41df8a82970d4edc87694e09b230adffb6cb39304cc70ab973616ca8500fbd1e" exitCode=60 Nov 03 22:15:04.133541 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.133474 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" event={"ID":"cd644ea6-af94-46b1-935f-019350f24563","Type":"ContainerDied","Data":"41df8a82970d4edc87694e09b230adffb6cb39304cc70ab973616ca8500fbd1e"} Nov 03 22:15:04.133738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.133721 5931 scope.go:122] "RemoveContainer" containerID="41df8a82970d4edc87694e09b230adffb6cb39304cc70ab973616ca8500fbd1e" Nov 03 22:15:04.133952 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:04.133937 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh_odh-ai-gateway-infra(cd644ea6-af94-46b1-935f-019350f24563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" podUID="cd644ea6-af94-46b1-935f-019350f24563" Nov 03 22:15:04.135638 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.135586 5931 generic.go:356] "Generic (PLEG): container finished" podID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerID="0d4c2fe710694ada68ba66c59365fdc2ede11ae0c058ed3e44ebc55c5afca8ce" exitCode=60 Nov 03 22:15:04.135738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.135700 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" event={"ID":"50ee43eb-927d-4a38-89f4-fcd2c0ca7087","Type":"ContainerDied","Data":"0d4c2fe710694ada68ba66c59365fdc2ede11ae0c058ed3e44ebc55c5afca8ce"} Nov 03 22:15:04.135886 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.135869 5931 scope.go:122] "RemoveContainer" containerID="0d4c2fe710694ada68ba66c59365fdc2ede11ae0c058ed3e44ebc55c5afca8ce" Nov 03 22:15:04.136102 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:04.136083 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp_odh-ai-gateway-infra(50ee43eb-927d-4a38-89f4-fcd2c0ca7087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" Nov 03 22:15:04.146728 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.146706 5931 scope.go:122] "RemoveContainer" containerID="e0af74504fcb380d57a459e88e7f1536f67f9271cbbb84d91eceaa22125095b3" Nov 03 22:15:04.161391 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.161375 5931 scope.go:122] "RemoveContainer" containerID="196f713d487f173bfcb6d5c2a8ff1006614b006cdbdaef62b1cb62e9a6b4d6ae" Nov 03 22:15:04.173304 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:04.173285 5931 scope.go:122] "RemoveContainer" containerID="17d82362fb3e687b6d92fca638937b8bffb218a1c95c69207e91975fcbd80e41" Nov 03 22:15:05.144183 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:05.144154 5931 scope.go:122] "RemoveContainer" containerID="41df8a82970d4edc87694e09b230adffb6cb39304cc70ab973616ca8500fbd1e" Nov 03 22:15:05.144672 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:05.144477 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh_odh-ai-gateway-infra(cd644ea6-af94-46b1-935f-019350f24563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" podUID="cd644ea6-af94-46b1-935f-019350f24563" Nov 03 22:15:05.145965 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:05.145941 5931 scope.go:122] "RemoveContainer" containerID="0d4c2fe710694ada68ba66c59365fdc2ede11ae0c058ed3e44ebc55c5afca8ce" Nov 03 22:15:05.146280 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:05.146245 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp_odh-ai-gateway-infra(50ee43eb-927d-4a38-89f4-fcd2c0ca7087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" Nov 03 22:15:05.147640 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:05.147602 5931 scope.go:122] "RemoveContainer" containerID="24f8967ddc1e109405ba19266ac90fbd403cddd5d0c81987923fd0f97735e3ca" Nov 03 22:15:05.147869 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:05.147850 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq_odh-ai-gateway-infra(87832f8b-cef0-42e9-afd0-98db0e82689f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" Nov 03 22:15:05.149456 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:05.149436 5931 scope.go:122] "RemoveContainer" containerID="37259d4b9731eae4e7c1f98867c27e127270c13f463161134782194b92101685" Nov 03 22:15:05.149680 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:05.149660 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l_odh-ai-gateway-infra(7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" Nov 03 22:15:06.831360 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:06.831325 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:08.287812 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:08.287762 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:08.544201 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:08.544110 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:11.988887 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:11.988853 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:12.267564 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:12.267485 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:12.620230 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:12.620199 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:13.330817 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:13.330783 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:13.688667 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:13.688606 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:15.499309 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:15.499277 5931 scope.go:122] "RemoveContainer" containerID="37259d4b9731eae4e7c1f98867c27e127270c13f463161134782194b92101685" Nov 03 22:15:16.238699 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:16.238669 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" event={"ID":"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75","Type":"ContainerStarted","Data":"e0d291b242d58c87562cc9a9b05d01906dd0611d9ec67538b0dfe10a31eccb47"} Nov 03 22:15:16.499499 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:16.499441 5931 scope.go:122] "RemoveContainer" containerID="41df8a82970d4edc87694e09b230adffb6cb39304cc70ab973616ca8500fbd1e" Nov 03 22:15:16.777257 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:16.777170 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" podUID="3f308068-3c70-4e1b-8fe9-04d2674b9ded" containerName="main" containerID="cri-o://8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca" gracePeriod=45 Nov 03 22:15:16.852581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:16.852544 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:17.035551 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.035497 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:15:17.164190 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164162 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tls-certs\") pod \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " Nov 03 22:15:17.164350 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164227 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-model-cache\") pod \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " Nov 03 22:15:17.164350 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164263 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-dshm\") pod \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " Nov 03 22:15:17.164350 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164329 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tmp-dir\") pod \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " Nov 03 22:15:17.164538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164365 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-home\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-home\") pod \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " Nov 03 22:15:17.164538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164402 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f308068-3c70-4e1b-8fe9-04d2674b9ded-kube-api-access-m72vd\" (UniqueName: \"kubernetes.io/projected/3f308068-3c70-4e1b-8fe9-04d2674b9ded-kube-api-access-m72vd\") pod \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\" (UID: \"3f308068-3c70-4e1b-8fe9-04d2674b9ded\") " Nov 03 22:15:17.164688 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164528 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-model-cache" pod "3f308068-3c70-4e1b-8fe9-04d2674b9ded" (UID: "3f308068-3c70-4e1b-8fe9-04d2674b9ded"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:17.164688 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164601 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tmp-dir" pod "3f308068-3c70-4e1b-8fe9-04d2674b9ded" (UID: "3f308068-3c70-4e1b-8fe9-04d2674b9ded"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:17.164805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.164722 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-home" pod "3f308068-3c70-4e1b-8fe9-04d2674b9ded" (UID: "3f308068-3c70-4e1b-8fe9-04d2674b9ded"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:17.165226 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.165207 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:17.165291 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.165231 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:17.165291 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.165246 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:17.166807 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.166778 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tls-certs" pod "3f308068-3c70-4e1b-8fe9-04d2674b9ded" (UID: "3f308068-3c70-4e1b-8fe9-04d2674b9ded"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:15:17.166947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.166933 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-dshm" pod "3f308068-3c70-4e1b-8fe9-04d2674b9ded" (UID: "3f308068-3c70-4e1b-8fe9-04d2674b9ded"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:17.167059 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.167043 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f308068-3c70-4e1b-8fe9-04d2674b9ded-kube-api-access-m72vd" pod "3f308068-3c70-4e1b-8fe9-04d2674b9ded" (UID: "3f308068-3c70-4e1b-8fe9-04d2674b9ded"). InnerVolumeSpecName "kube-api-access-m72vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:17.250440 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.250412 5931 generic.go:356] "Generic (PLEG): container finished" podID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerID="e0d291b242d58c87562cc9a9b05d01906dd0611d9ec67538b0dfe10a31eccb47" exitCode=60 Nov 03 22:15:17.250567 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.250486 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" event={"ID":"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75","Type":"ContainerDied","Data":"e0d291b242d58c87562cc9a9b05d01906dd0611d9ec67538b0dfe10a31eccb47"} Nov 03 22:15:17.250567 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.250545 5931 scope.go:122] "RemoveContainer" containerID="37259d4b9731eae4e7c1f98867c27e127270c13f463161134782194b92101685" Nov 03 22:15:17.250924 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.250884 5931 scope.go:122] "RemoveContainer" containerID="e0d291b242d58c87562cc9a9b05d01906dd0611d9ec67538b0dfe10a31eccb47" Nov 03 22:15:17.251207 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:17.251182 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l_odh-ai-gateway-infra(7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" Nov 03 22:15:17.252524 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.252500 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" event={"ID":"cd644ea6-af94-46b1-935f-019350f24563","Type":"ContainerStarted","Data":"bb4cc1152ca49e07e4a0b3c0fe8cf34a66c0b8db2c1d30115a45d61df40b356b"} Nov 03 22:15:17.253746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.253728 5931 generic.go:356] "Generic (PLEG): container finished" podID="3f308068-3c70-4e1b-8fe9-04d2674b9ded" containerID="8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca" exitCode=0 Nov 03 22:15:17.253834 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.253764 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" Nov 03 22:15:17.253834 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.253808 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" event={"ID":"3f308068-3c70-4e1b-8fe9-04d2674b9ded","Type":"ContainerDied","Data":"8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca"} Nov 03 22:15:17.253909 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.253837 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s" event={"ID":"3f308068-3c70-4e1b-8fe9-04d2674b9ded","Type":"ContainerDied","Data":"ed2ab3144617bb2f6938ba063d6595bfdc0d5553772bb7ef08a4f53c1b780e96"} Nov 03 22:15:17.263606 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.263585 5931 scope.go:122] "RemoveContainer" containerID="8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca" Nov 03 22:15:17.266127 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.266110 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f308068-3c70-4e1b-8fe9-04d2674b9ded-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:17.266202 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.266130 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f308068-3c70-4e1b-8fe9-04d2674b9ded-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:17.266202 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.266142 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m72vd\" (UniqueName: \"kubernetes.io/projected/3f308068-3c70-4e1b-8fe9-04d2674b9ded-kube-api-access-m72vd\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:17.272500 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.272483 5931 scope.go:122] "RemoveContainer" containerID="8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca" Nov 03 22:15:17.272774 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:17.272753 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca\": container with ID starting with 8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca not found: ID does not exist" containerID="8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca" Nov 03 22:15:17.272828 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.272781 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca"} err="failed to get container status \"8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca\": rpc error: code = NotFound desc = could not find container \"8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca\": container with ID starting with 8aa20ada3f10bc2ea92b2598b86afd4a257362568f2fa3ee04cd32cf3e4ddfca not found: ID does not exist" Nov 03 22:15:17.363115 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.363083 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l"] Nov 03 22:15:17.484983 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.484954 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s"] Nov 03 22:15:17.488199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.488174 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a49a32/auth-test-model-a49a32-kserve-86c9dd4b45-v6q5s"] Nov 03 22:15:17.499138 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.499121 5931 scope.go:122] "RemoveContainer" containerID="0d4c2fe710694ada68ba66c59365fdc2ede11ae0c058ed3e44ebc55c5afca8ce" Nov 03 22:15:17.499221 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.499210 5931 scope.go:122] "RemoveContainer" containerID="24f8967ddc1e109405ba19266ac90fbd403cddd5d0c81987923fd0f97735e3ca" Nov 03 22:15:17.719577 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.719494 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:17.956278 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:17.956253 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:18.262188 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.262155 5931 generic.go:356] "Generic (PLEG): container finished" podID="cd644ea6-af94-46b1-935f-019350f24563" containerID="bb4cc1152ca49e07e4a0b3c0fe8cf34a66c0b8db2c1d30115a45d61df40b356b" exitCode=60 Nov 03 22:15:18.262324 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.262190 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" event={"ID":"cd644ea6-af94-46b1-935f-019350f24563","Type":"ContainerDied","Data":"bb4cc1152ca49e07e4a0b3c0fe8cf34a66c0b8db2c1d30115a45d61df40b356b"} Nov 03 22:15:18.262324 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.262251 5931 scope.go:122] "RemoveContainer" containerID="41df8a82970d4edc87694e09b230adffb6cb39304cc70ab973616ca8500fbd1e" Nov 03 22:15:18.262600 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.262581 5931 scope.go:122] "RemoveContainer" containerID="bb4cc1152ca49e07e4a0b3c0fe8cf34a66c0b8db2c1d30115a45d61df40b356b" Nov 03 22:15:18.262903 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:18.262878 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh_odh-ai-gateway-infra(cd644ea6-af94-46b1-935f-019350f24563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" podUID="cd644ea6-af94-46b1-935f-019350f24563" Nov 03 22:15:18.265436 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.265413 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" event={"ID":"50ee43eb-927d-4a38-89f4-fcd2c0ca7087","Type":"ContainerStarted","Data":"a0a3018c17f9576145885917baf0d19e998926410eef21b51808f83f296bdac5"} Nov 03 22:15:18.267461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.267439 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" event={"ID":"87832f8b-cef0-42e9-afd0-98db0e82689f","Type":"ContainerStarted","Data":"19f52ec7710796662b9f3defc709c37c2f5ff950c29035c7a04107373a974ccd"} Nov 03 22:15:18.449192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.449161 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:18.507806 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.507776 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f308068-3c70-4e1b-8fe9-04d2674b9ded" path="/var/lib/kubelet/pods/3f308068-3c70-4e1b-8fe9-04d2674b9ded/volumes" Nov 03 22:15:18.586973 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.586912 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-kube-api-access-fcb2z\" (UniqueName: \"kubernetes.io/projected/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-kube-api-access-fcb2z\") pod \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " Nov 03 22:15:18.587113 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.587101 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-maas-api-service-ca\") pod \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\" (UID: \"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75\") " Nov 03 22:15:18.587517 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.587448 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-maas-api-service-ca" pod "7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" (UID: "7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:18.588913 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.588889 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-kube-api-access-fcb2z" pod "7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" (UID: "7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75"). InnerVolumeSpecName "kube-api-access-fcb2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:18.606365 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.606344 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh"] Nov 03 22:15:18.688384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.688354 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-maas-api-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:18.688384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:18.688381 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcb2z\" (UniqueName: \"kubernetes.io/projected/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75-kube-api-access-fcb2z\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:19.279720 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.279689 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" Nov 03 22:15:19.280182 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.279720 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l" event={"ID":"7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75","Type":"ContainerDied","Data":"b75f7915590e255e367c4a48a990536d693e2789e1ef99f3697dce5b5707fff2"} Nov 03 22:15:19.280182 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.279798 5931 scope.go:122] "RemoveContainer" containerID="e0d291b242d58c87562cc9a9b05d01906dd0611d9ec67538b0dfe10a31eccb47" Nov 03 22:15:19.295068 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.295028 5931 generic.go:356] "Generic (PLEG): container finished" podID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerID="a0a3018c17f9576145885917baf0d19e998926410eef21b51808f83f296bdac5" exitCode=60 Nov 03 22:15:19.295197 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.295090 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" event={"ID":"50ee43eb-927d-4a38-89f4-fcd2c0ca7087","Type":"ContainerDied","Data":"a0a3018c17f9576145885917baf0d19e998926410eef21b51808f83f296bdac5"} Nov 03 22:15:19.295512 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.295488 5931 scope.go:122] "RemoveContainer" containerID="a0a3018c17f9576145885917baf0d19e998926410eef21b51808f83f296bdac5" Nov 03 22:15:19.295887 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:19.295782 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp_odh-ai-gateway-infra(50ee43eb-927d-4a38-89f4-fcd2c0ca7087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" Nov 03 22:15:19.297469 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.297445 5931 generic.go:356] "Generic (PLEG): container finished" podID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerID="19f52ec7710796662b9f3defc709c37c2f5ff950c29035c7a04107373a974ccd" exitCode=60 Nov 03 22:15:19.297577 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.297522 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" event={"ID":"87832f8b-cef0-42e9-afd0-98db0e82689f","Type":"ContainerDied","Data":"19f52ec7710796662b9f3defc709c37c2f5ff950c29035c7a04107373a974ccd"} Nov 03 22:15:19.297956 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.297937 5931 scope.go:122] "RemoveContainer" containerID="19f52ec7710796662b9f3defc709c37c2f5ff950c29035c7a04107373a974ccd" Nov 03 22:15:19.298206 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:19.298184 5931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq_odh-ai-gateway-infra(87832f8b-cef0-42e9-afd0-98db0e82689f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" Nov 03 22:15:19.299911 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.299881 5931 scope.go:122] "RemoveContainer" containerID="0d4c2fe710694ada68ba66c59365fdc2ede11ae0c058ed3e44ebc55c5afca8ce" Nov 03 22:15:19.313781 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.313764 5931 scope.go:122] "RemoveContainer" containerID="24f8967ddc1e109405ba19266ac90fbd403cddd5d0c81987923fd0f97735e3ca" Nov 03 22:15:19.439246 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.439219 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:19.508563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.508539 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq"] Nov 03 22:15:19.554093 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.554036 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l"] Nov 03 22:15:19.558028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.558003 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-29dbd9-29895735-xtw6l"] Nov 03 22:15:19.597928 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.597903 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd644ea6-af94-46b1-935f-019350f24563-kube-api-access-m2cvl\" (UniqueName: \"kubernetes.io/projected/cd644ea6-af94-46b1-935f-019350f24563-kube-api-access-m2cvl\") pod \"cd644ea6-af94-46b1-935f-019350f24563\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " Nov 03 22:15:19.598044 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.597954 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd644ea6-af94-46b1-935f-019350f24563-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd644ea6-af94-46b1-935f-019350f24563-maas-api-service-ca\") pod \"cd644ea6-af94-46b1-935f-019350f24563\" (UID: \"cd644ea6-af94-46b1-935f-019350f24563\") " Nov 03 22:15:19.598360 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.598331 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd644ea6-af94-46b1-935f-019350f24563-maas-api-service-ca" pod "cd644ea6-af94-46b1-935f-019350f24563" (UID: "cd644ea6-af94-46b1-935f-019350f24563"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:19.600358 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.600333 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd644ea6-af94-46b1-935f-019350f24563-kube-api-access-m2cvl" pod "cd644ea6-af94-46b1-935f-019350f24563" (UID: "cd644ea6-af94-46b1-935f-019350f24563"). InnerVolumeSpecName "kube-api-access-m2cvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:19.620281 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.620261 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp"] Nov 03 22:15:19.699257 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.699237 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2cvl\" (UniqueName: \"kubernetes.io/projected/cd644ea6-af94-46b1-935f-019350f24563-kube-api-access-m2cvl\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:19.699257 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:19.699259 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd644ea6-af94-46b1-935f-019350f24563-maas-api-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:20.306288 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.306250 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" event={"ID":"cd644ea6-af94-46b1-935f-019350f24563","Type":"ContainerDied","Data":"5e91b83a3997d507ab3659405bc2cdf3949df6dbb8e783bce0fd37676492311e"} Nov 03 22:15:20.306734 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.306263 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh" Nov 03 22:15:20.306734 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.306367 5931 scope.go:122] "RemoveContainer" containerID="bb4cc1152ca49e07e4a0b3c0fe8cf34a66c0b8db2c1d30115a45d61df40b356b" Nov 03 22:15:20.459049 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.459023 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:20.472962 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.472932 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh"] Nov 03 22:15:20.480393 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.480361 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75f577-29895735-9fpqh"] Nov 03 22:15:20.494391 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.494368 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:20.506740 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.506715 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" path="/var/lib/kubelet/pods/7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75/volumes" Nov 03 22:15:20.507171 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.507155 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd644ea6-af94-46b1-935f-019350f24563" path="/var/lib/kubelet/pods/cd644ea6-af94-46b1-935f-019350f24563/volumes" Nov 03 22:15:20.507746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.507693 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87832f8b-cef0-42e9-afd0-98db0e82689f-kube-api-access-hbvjb\" (UniqueName: \"kubernetes.io/projected/87832f8b-cef0-42e9-afd0-98db0e82689f-kube-api-access-hbvjb\") pod \"87832f8b-cef0-42e9-afd0-98db0e82689f\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " Nov 03 22:15:20.508066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.508042 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-kube-api-access-8hg8h\" (UniqueName: \"kubernetes.io/projected/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-kube-api-access-8hg8h\") pod \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " Nov 03 22:15:20.508305 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.508281 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-maas-api-service-ca\") pod \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\" (UID: \"50ee43eb-927d-4a38-89f4-fcd2c0ca7087\") " Nov 03 22:15:20.508451 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.508315 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87832f8b-cef0-42e9-afd0-98db0e82689f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87832f8b-cef0-42e9-afd0-98db0e82689f-maas-api-service-ca\") pod \"87832f8b-cef0-42e9-afd0-98db0e82689f\" (UID: \"87832f8b-cef0-42e9-afd0-98db0e82689f\") " Nov 03 22:15:20.508825 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.508778 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87832f8b-cef0-42e9-afd0-98db0e82689f-maas-api-service-ca" pod "87832f8b-cef0-42e9-afd0-98db0e82689f" (UID: "87832f8b-cef0-42e9-afd0-98db0e82689f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:20.508984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.508948 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-maas-api-service-ca" pod "50ee43eb-927d-4a38-89f4-fcd2c0ca7087" (UID: "50ee43eb-927d-4a38-89f4-fcd2c0ca7087"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:20.510288 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.510266 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87832f8b-cef0-42e9-afd0-98db0e82689f-kube-api-access-hbvjb" pod "87832f8b-cef0-42e9-afd0-98db0e82689f" (UID: "87832f8b-cef0-42e9-afd0-98db0e82689f"). InnerVolumeSpecName "kube-api-access-hbvjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:20.510550 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.510532 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-kube-api-access-8hg8h" pod "50ee43eb-927d-4a38-89f4-fcd2c0ca7087" (UID: "50ee43eb-927d-4a38-89f4-fcd2c0ca7087"). InnerVolumeSpecName "kube-api-access-8hg8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:20.610195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.610165 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hg8h\" (UniqueName: \"kubernetes.io/projected/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-kube-api-access-8hg8h\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:20.610195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.610192 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ee43eb-927d-4a38-89f4-fcd2c0ca7087-maas-api-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:20.610195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.610201 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87832f8b-cef0-42e9-afd0-98db0e82689f-maas-api-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:20.610414 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.610210 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvjb\" (UniqueName: \"kubernetes.io/projected/87832f8b-cef0-42e9-afd0-98db0e82689f-kube-api-access-hbvjb\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:20.995191 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:20.995112 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:21.231957 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.231928 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:21.302877 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.302797 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9"] Nov 03 22:15:21.303384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303371 5931 cpu_manager.go:414] "containerMap: removing container" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303386 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303394 5931 cpu_manager.go:414] "containerMap: removing container" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303399 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303410 5931 cpu_manager.go:414] "containerMap: removing container" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.303430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303414 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.303430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303421 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.303430 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303426 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303434 5931 cpu_manager.go:414] "containerMap: removing container" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303438 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303458 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303462 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303477 5931 cpu_manager.go:414] "containerMap: removing container" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303481 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303488 5931 cpu_manager.go:414] "containerMap: removing container" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303492 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303502 5931 cpu_manager.go:414] "containerMap: removing container" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303506 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303513 5931 cpu_manager.go:414] "containerMap: removing container" podUID="3f308068-3c70-4e1b-8fe9-04d2674b9ded" containerName="main" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303517 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f308068-3c70-4e1b-8fe9-04d2674b9ded" containerName="main" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303598 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303619 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303627 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303632 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303638 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303642 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303649 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303657 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303663 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303669 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.303672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.303676 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f308068-3c70-4e1b-8fe9-04d2674b9ded" containerName="main" Nov 03 22:15:21.308203 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.308188 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.318311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.318290 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.318458 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.318339 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vls\" (UniqueName: \"kubernetes.io/projected/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-kube-api-access-f2vls\") pod \"payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.322757 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.322735 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" Nov 03 22:15:21.322850 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.322772 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp" event={"ID":"50ee43eb-927d-4a38-89f4-fcd2c0ca7087","Type":"ContainerDied","Data":"0a931d4e2d61d8a8aaba0a5c63bb1c3803890f8d99527d36f0c976e137934d8f"} Nov 03 22:15:21.322850 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.322830 5931 scope.go:122] "RemoveContainer" containerID="a0a3018c17f9576145885917baf0d19e998926410eef21b51808f83f296bdac5" Nov 03 22:15:21.324335 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.324319 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" Nov 03 22:15:21.324436 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.324357 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq" event={"ID":"87832f8b-cef0-42e9-afd0-98db0e82689f","Type":"ContainerDied","Data":"131109c6b1157bb1851a23c5eaf83f9b5ac0cad981071835d3d45abede37643e"} Nov 03 22:15:21.334307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.334291 5931 scope.go:122] "RemoveContainer" containerID="19f52ec7710796662b9f3defc709c37c2f5ff950c29035c7a04107373a974ccd" Nov 03 22:15:21.361182 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.361149 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-534c77-dockercfg-vf979\"" Nov 03 22:15:21.364737 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.364713 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9"] Nov 03 22:15:21.409380 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.409350 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-534c77\"" Nov 03 22:15:21.420568 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.420534 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.420745 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.420629 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vls\" (UniqueName: \"kubernetes.io/projected/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-kube-api-access-f2vls\") pod \"payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.421471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.421446 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.448513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.448463 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk"] Nov 03 22:15:21.449190 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449169 5931 cpu_manager.go:414] "containerMap: removing container" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.449190 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449188 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.449325 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449211 5931 cpu_manager.go:414] "containerMap: removing container" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.449325 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449215 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" containerName="cleanup" Nov 03 22:15:21.449325 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449224 5931 cpu_manager.go:414] "containerMap: removing container" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.449325 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449229 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd644ea6-af94-46b1-935f-019350f24563" containerName="cleanup" Nov 03 22:15:21.449452 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449341 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" containerName="cleanup" Nov 03 22:15:21.449452 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.449353 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af1c38c-cdf7-4c99-ab1c-dae3d5ba7b75" containerName="cleanup" Nov 03 22:15:21.454238 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.454222 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:21.477272 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.477249 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vls\" (UniqueName: \"kubernetes.io/projected/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-kube-api-access-f2vls\") pod \"payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.509779 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.509756 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk"] Nov 03 22:15:21.522521 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.522491 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d690c88c-dbed-4959-9c2c-722a6f41e8b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:21.522713 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.522692 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77xzs\" (UniqueName: \"kubernetes.io/projected/d690c88c-dbed-4959-9c2c-722a6f41e8b6-kube-api-access-77xzs\") pod \"payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:21.535173 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.535141 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp"] Nov 03 22:15:21.537798 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.537780 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895735-bsnsp"] Nov 03 22:15:21.623345 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.623324 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:21.623813 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.623792 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d690c88c-dbed-4959-9c2c-722a6f41e8b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:21.623887 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.623862 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77xzs\" (UniqueName: \"kubernetes.io/projected/d690c88c-dbed-4959-9c2c-722a6f41e8b6-kube-api-access-77xzs\") pod \"payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:21.624444 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.624421 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d690c88c-dbed-4959-9c2c-722a6f41e8b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:21.645317 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.645294 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq"] Nov 03 22:15:21.649799 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.649777 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ddbba-29895735-4brkq"] Nov 03 22:15:21.682101 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.682051 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77xzs\" (UniqueName: \"kubernetes.io/projected/d690c88c-dbed-4959-9c2c-722a6f41e8b6-kube-api-access-77xzs\") pod \"payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:21.768122 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:21.768098 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:22.005850 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.005820 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9"] Nov 03 22:15:22.143987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.143963 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk"] Nov 03 22:15:22.145631 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:15:22.145582 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd690c88c_dbed_4959_9c2c_722a6f41e8b6.slice/crio-941208d6dbb8d2885577273bc678bd8aa198ddb0811579f7b068950158e32e69 WatchSource:0}: Error finding container 941208d6dbb8d2885577273bc678bd8aa198ddb0811579f7b068950158e32e69: Status 404 returned error can't find the container with id 941208d6dbb8d2885577273bc678bd8aa198ddb0811579f7b068950158e32e69 Nov 03 22:15:22.184847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.184820 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:22.332973 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.332897 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" event={"ID":"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb","Type":"ContainerStarted","Data":"fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02"} Nov 03 22:15:22.332973 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.332946 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" event={"ID":"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb","Type":"ContainerStarted","Data":"1cf31910cacabe9b2f974ba5aca6a18e38604502658ca7f56d79bd7e03a43bc2"} Nov 03 22:15:22.333408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.333007 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:22.336217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.336188 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" event={"ID":"d690c88c-dbed-4959-9c2c-722a6f41e8b6","Type":"ContainerStarted","Data":"e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595"} Nov 03 22:15:22.336348 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.336246 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:22.336348 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.336270 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" event={"ID":"d690c88c-dbed-4959-9c2c-722a6f41e8b6","Type":"ContainerStarted","Data":"941208d6dbb8d2885577273bc678bd8aa198ddb0811579f7b068950158e32e69"} Nov 03 22:15:22.446155 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.446110 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" podStartSLOduration=1.446095203 podStartE2EDuration="1.446095203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:22.445493698 +0000 UTC m=+3590.056341513" watchObservedRunningTime="2026-11-03 22:15:22.446095203 +0000 UTC m=+3590.056943019" Nov 03 22:15:22.507495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.507472 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ee43eb-927d-4a38-89f4-fcd2c0ca7087" path="/var/lib/kubelet/pods/50ee43eb-927d-4a38-89f4-fcd2c0ca7087/volumes" Nov 03 22:15:22.507929 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.507912 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87832f8b-cef0-42e9-afd0-98db0e82689f" path="/var/lib/kubelet/pods/87832f8b-cef0-42e9-afd0-98db0e82689f/volumes" Nov 03 22:15:22.559994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:22.559944 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" podStartSLOduration=1.559927442 podStartE2EDuration="1.559927442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:22.557570239 +0000 UTC m=+3590.168418053" watchObservedRunningTime="2026-11-03 22:15:22.559927442 +0000 UTC m=+3590.170775260" Nov 03 22:15:27.267736 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:27.267702 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:29.071235 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:29.071207 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7"] Nov 03 22:15:29.175871 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:29.175841 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb"] Nov 03 22:15:29.539922 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:29.539888 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:29.773872 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:29.773843 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:32.018858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:32.018823 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:33.067675 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:33.067639 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:15:33.348964 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:33.348931 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:15:33.349122 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:33.349015 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:15:34.072238 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.072198 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" podUID="5e5428f9-9a24-458a-80a1-1c23be765909" containerName="payload-processing" containerID="cri-o://e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29" gracePeriod=25 Nov 03 22:15:34.115686 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.115654 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:34.177056 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.177029 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" podUID="5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" containerName="payload-pre-processing" containerID="cri-o://cc0efffad83b007d3c128ef699591935f39ae213a68e8650176c5193ecf4bb2e" gracePeriod=25 Nov 03 22:15:34.345465 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.345440 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:15:34.433486 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.433446 5931 generic.go:356] "Generic (PLEG): container finished" podID="5e5428f9-9a24-458a-80a1-1c23be765909" containerID="e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29" exitCode=0 Nov 03 22:15:34.433668 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.433503 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" Nov 03 22:15:34.433668 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.433524 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" event={"ID":"5e5428f9-9a24-458a-80a1-1c23be765909","Type":"ContainerDied","Data":"e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29"} Nov 03 22:15:34.433668 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.433601 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7" event={"ID":"5e5428f9-9a24-458a-80a1-1c23be765909","Type":"ContainerDied","Data":"4926e800960e6c74fc37d597f94642a4c98453cb2b43c7d486df4af01fe1b118"} Nov 03 22:15:34.433668 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.433661 5931 scope.go:122] "RemoveContainer" containerID="e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29" Nov 03 22:15:34.435265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.435244 5931 generic.go:356] "Generic (PLEG): container finished" podID="5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" containerID="cc0efffad83b007d3c128ef699591935f39ae213a68e8650176c5193ecf4bb2e" exitCode=0 Nov 03 22:15:34.435346 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.435313 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" event={"ID":"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6","Type":"ContainerDied","Data":"cc0efffad83b007d3c128ef699591935f39ae213a68e8650176c5193ecf4bb2e"} Nov 03 22:15:34.447226 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.447209 5931 scope.go:122] "RemoveContainer" containerID="e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29" Nov 03 22:15:34.447494 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:34.447475 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29\": container with ID starting with e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29 not found: ID does not exist" containerID="e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29" Nov 03 22:15:34.447562 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.447505 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29"} err="failed to get container status \"e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29\": rpc error: code = NotFound desc = could not find container \"e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29\": container with ID starting with e80795f7fd86a9e62487a5f34c4598ce99826ca936ebcb5357923744097cfa29 not found: ID does not exist" Nov 03 22:15:34.448619 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.448590 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5428f9-9a24-458a-80a1-1c23be765909-kube-api-access-82x2b\" (UniqueName: \"kubernetes.io/projected/5e5428f9-9a24-458a-80a1-1c23be765909-kube-api-access-82x2b\") pod \"5e5428f9-9a24-458a-80a1-1c23be765909\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " Nov 03 22:15:34.448665 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.448650 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5428f9-9a24-458a-80a1-1c23be765909-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5428f9-9a24-458a-80a1-1c23be765909-plugins-config-volume\") pod \"5e5428f9-9a24-458a-80a1-1c23be765909\" (UID: \"5e5428f9-9a24-458a-80a1-1c23be765909\") " Nov 03 22:15:34.449003 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.448984 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5428f9-9a24-458a-80a1-1c23be765909-plugins-config-volume" pod "5e5428f9-9a24-458a-80a1-1c23be765909" (UID: "5e5428f9-9a24-458a-80a1-1c23be765909"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:34.449342 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.449282 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5428f9-9a24-458a-80a1-1c23be765909-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:34.449458 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.449441 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:15:34.450733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.450710 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5428f9-9a24-458a-80a1-1c23be765909-kube-api-access-82x2b" pod "5e5428f9-9a24-458a-80a1-1c23be765909" (UID: "5e5428f9-9a24-458a-80a1-1c23be765909"). InnerVolumeSpecName "kube-api-access-82x2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:34.550519 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.550490 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-plugins-config-volume\") pod \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " Nov 03 22:15:34.550698 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.550682 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-kube-api-access-xkv7j\" (UniqueName: \"kubernetes.io/projected/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-kube-api-access-xkv7j\") pod \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\" (UID: \"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6\") " Nov 03 22:15:34.550852 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.550832 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-plugins-config-volume" pod "5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" (UID: "5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:34.551111 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.551094 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:34.551199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.551113 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82x2b\" (UniqueName: \"kubernetes.io/projected/5e5428f9-9a24-458a-80a1-1c23be765909-kube-api-access-82x2b\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:34.552794 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.552777 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-kube-api-access-xkv7j" pod "5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" (UID: "5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6"). InnerVolumeSpecName "kube-api-access-xkv7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:34.632882 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.632809 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:34.652439 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.652413 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkv7j\" (UniqueName: \"kubernetes.io/projected/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6-kube-api-access-xkv7j\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:34.862470 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.862440 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7"] Nov 03 22:15:34.880091 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:34.880062 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-12dee4-c4cd4686-mqfq7"] Nov 03 22:15:35.139093 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.139067 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:35.443720 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.443695 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" Nov 03 22:15:35.443884 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.443720 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb" event={"ID":"5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6","Type":"ContainerDied","Data":"8410e6bf1840426b3152ecd5e2d309f0c9eb4134597d8c241ef16abf41f33e61"} Nov 03 22:15:35.443884 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.443773 5931 scope.go:122] "RemoveContainer" containerID="cc0efffad83b007d3c128ef699591935f39ae213a68e8650176c5193ecf4bb2e" Nov 03 22:15:35.608333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.608305 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb"] Nov 03 22:15:35.657233 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.657207 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-12dee4-767c5b7959-kh2hb"] Nov 03 22:15:35.829948 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.829890 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:15:35.836417 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:35.836398 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:15:36.506330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:36.506296 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" path="/var/lib/kubelet/pods/5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6/volumes" Nov 03 22:15:36.506698 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:36.506679 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5428f9-9a24-458a-80a1-1c23be765909" path="/var/lib/kubelet/pods/5e5428f9-9a24-458a-80a1-1c23be765909/volumes" Nov 03 22:15:36.705228 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:36.705196 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:39.630486 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:39.630458 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:39.865211 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:39.865182 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:40.460242 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:40.460210 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27"] Nov 03 22:15:40.566678 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:40.566603 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:41.786483 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:41.786404 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:42.009698 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:42.009660 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:45.315310 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.315267 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:45.543829 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.543796 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:45.597328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.597297 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q"] Nov 03 22:15:45.598031 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.598012 5931 cpu_manager.go:414] "containerMap: removing container" podUID="5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" containerName="payload-pre-processing" Nov 03 22:15:45.598158 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.598032 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" containerName="payload-pre-processing" Nov 03 22:15:45.598158 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.598061 5931 cpu_manager.go:414] "containerMap: removing container" podUID="5e5428f9-9a24-458a-80a1-1c23be765909" containerName="payload-processing" Nov 03 22:15:45.598158 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.598068 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5428f9-9a24-458a-80a1-1c23be765909" containerName="payload-processing" Nov 03 22:15:45.598323 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.598223 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7a8c70-8f5c-46fc-bd8d-e91cd9f59bb6" containerName="payload-pre-processing" Nov 03 22:15:45.598323 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.598258 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5428f9-9a24-458a-80a1-1c23be765909" containerName="payload-processing" Nov 03 22:15:45.603538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.603518 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:45.658129 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.658098 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q"] Nov 03 22:15:45.702937 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.702914 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d7b12e\"" Nov 03 22:15:45.748793 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.748756 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r"] Nov 03 22:15:45.755325 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.755298 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:45.770859 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.770818 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a523e83-dfb0-42a8-ac6a-99d640044275-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:45.771007 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.770916 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2jv\" (UniqueName: \"kubernetes.io/projected/8a523e83-dfb0-42a8-ac6a-99d640044275-kube-api-access-jw2jv\") pod \"payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:45.808992 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.808967 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d7b12e-dockercfg-lxjsn\"" Nov 03 22:15:45.820086 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.820053 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r"] Nov 03 22:15:45.871990 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.871925 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a523e83-dfb0-42a8-ac6a-99d640044275-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:45.871990 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.871987 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2jv\" (UniqueName: \"kubernetes.io/projected/8a523e83-dfb0-42a8-ac6a-99d640044275-kube-api-access-jw2jv\") pod \"payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:45.872146 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.872116 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392dfab9-5b10-4ea9-b752-2dbedecfabea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:45.872181 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.872161 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9n86\" (UniqueName: \"kubernetes.io/projected/392dfab9-5b10-4ea9-b752-2dbedecfabea-kube-api-access-r9n86\") pod \"payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:45.872600 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.872573 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a523e83-dfb0-42a8-ac6a-99d640044275-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:45.938741 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.938710 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw"] Nov 03 22:15:45.938996 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.938950 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" containerID="cri-o://5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1" gracePeriod=30 Nov 03 22:15:45.941434 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.941408 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2jv\" (UniqueName: \"kubernetes.io/projected/8a523e83-dfb0-42a8-ac6a-99d640044275-kube-api-access-jw2jv\") pod \"payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:45.973251 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.973224 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392dfab9-5b10-4ea9-b752-2dbedecfabea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:45.973372 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.973267 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9n86\" (UniqueName: \"kubernetes.io/projected/392dfab9-5b10-4ea9-b752-2dbedecfabea-kube-api-access-r9n86\") pod \"payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:45.973925 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:45.973902 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392dfab9-5b10-4ea9-b752-2dbedecfabea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:46.029307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:46.029287 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9n86\" (UniqueName: \"kubernetes.io/projected/392dfab9-5b10-4ea9-b752-2dbedecfabea-kube-api-access-r9n86\") pod \"payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:46.070491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:46.070468 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:46.216633 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:46.216589 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:46.353984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:46.353958 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:46.450906 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:46.450880 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r"] Nov 03 22:15:46.452704 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:15:46.452677 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392dfab9_5b10_4ea9_b752_2dbedecfabea.slice/crio-5980f4a787486329979d1d279a039d1d54127cc854712437132e2d64dbe1b78f WatchSource:0}: Error finding container 5980f4a787486329979d1d279a039d1d54127cc854712437132e2d64dbe1b78f: Status 404 returned error can't find the container with id 5980f4a787486329979d1d279a039d1d54127cc854712437132e2d64dbe1b78f Nov 03 22:15:46.521993 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:46.521963 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" event={"ID":"392dfab9-5b10-4ea9-b752-2dbedecfabea","Type":"ContainerStarted","Data":"5980f4a787486329979d1d279a039d1d54127cc854712437132e2d64dbe1b78f"} Nov 03 22:15:46.592267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:46.592226 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q"] Nov 03 22:15:46.597465 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:15:46.597434 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a523e83_dfb0_42a8_ac6a_99d640044275.slice/crio-95d35c64b14a195d7efaba80847b59db6359367010a08ec81ab539031a5d0bc0 WatchSource:0}: Error finding container 95d35c64b14a195d7efaba80847b59db6359367010a08ec81ab539031a5d0bc0: Status 404 returned error can't find the container with id 95d35c64b14a195d7efaba80847b59db6359367010a08ec81ab539031a5d0bc0 Nov 03 22:15:47.530440 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.530406 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" event={"ID":"392dfab9-5b10-4ea9-b752-2dbedecfabea","Type":"ContainerStarted","Data":"dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72"} Nov 03 22:15:47.530872 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.530478 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:47.532103 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.532077 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" event={"ID":"8a523e83-dfb0-42a8-ac6a-99d640044275","Type":"ContainerStarted","Data":"095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71"} Nov 03 22:15:47.532209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.532123 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" event={"ID":"8a523e83-dfb0-42a8-ac6a-99d640044275","Type":"ContainerStarted","Data":"95d35c64b14a195d7efaba80847b59db6359367010a08ec81ab539031a5d0bc0"} Nov 03 22:15:47.532209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.532150 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:47.643039 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.642791 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" podStartSLOduration=2.642775201 podStartE2EDuration="2.642775201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:47.642675911 +0000 UTC m=+3615.253523727" watchObservedRunningTime="2026-11-03 22:15:47.642775201 +0000 UTC m=+3615.253623015" Nov 03 22:15:47.756215 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.756160 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" podStartSLOduration=2.7561430639999998 podStartE2EDuration="2.756143064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:47.754455 +0000 UTC m=+3615.365302815" watchObservedRunningTime="2026-11-03 22:15:47.756143064 +0000 UTC m=+3615.366990879" Nov 03 22:15:47.911054 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:47.911021 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:48.163692 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:48.163599 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:48.294685 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:48.294647 5931 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:15:48.294854 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:48.294699 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:15:48.399180 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:48.399147 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:48.833665 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:48.833606 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:51.198074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.198048 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:15:51.329382 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329328 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-credential-socket\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329382 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329360 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-envoy\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329540 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329396 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-kube-api-access-bqkn7\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-kube-api-access-bqkn7\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329540 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329416 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-token\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329540 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329445 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-podinfo\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329533 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-data\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329578 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-socket\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329586 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-credential-socket" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:51.329733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329647 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-certs\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329940 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329740 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb4ab36-e83f-4782-a119-b50938bcf669-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4ab36-e83f-4782-a119-b50938bcf669-istiod-ca-cert\") pod \"9cb4ab36-e83f-4782-a119-b50938bcf669\" (UID: \"9cb4ab36-e83f-4782-a119-b50938bcf669\") " Nov 03 22:15:51.329940 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329908 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-data" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:51.329940 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329923 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-certs" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:51.330101 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.329931 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-socket" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:51.330257 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.330237 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb4ab36-e83f-4782-a119-b50938bcf669-istiod-ca-cert" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:15:51.330477 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.330452 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.330744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.330495 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.330744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.330521 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.330744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.330535 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.330744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.330549 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4ab36-e83f-4782-a119-b50938bcf669-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.331784 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.331763 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-kube-api-access-bqkn7" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "kube-api-access-bqkn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:51.331885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.331830 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-envoy" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:51.332138 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.332113 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-token" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:51.332224 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.332197 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-podinfo" pod "9cb4ab36-e83f-4782-a119-b50938bcf669" (UID: "9cb4ab36-e83f-4782-a119-b50938bcf669"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:15:51.431327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.431303 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.431327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.431324 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqkn7\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-kube-api-access-bqkn7\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.431479 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.431334 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.431479 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.431342 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4ab36-e83f-4782-a119-b50938bcf669-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:51.559333 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.559305 5931 generic.go:356] "Generic (PLEG): container finished" podID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerID="5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1" exitCode=0 Nov 03 22:15:51.559462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.559345 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" Nov 03 22:15:51.559462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.559353 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" event={"ID":"9cb4ab36-e83f-4782-a119-b50938bcf669","Type":"ContainerDied","Data":"5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1"} Nov 03 22:15:51.559462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.559388 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw" event={"ID":"9cb4ab36-e83f-4782-a119-b50938bcf669","Type":"ContainerDied","Data":"a9646d4d0967c5cfffe5a047b45e20810815de719ed099d5f7ca4b58ab998319"} Nov 03 22:15:51.559462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.559411 5931 scope.go:122] "RemoveContainer" containerID="5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1" Nov 03 22:15:51.570531 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.570514 5931 scope.go:122] "RemoveContainer" containerID="5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1" Nov 03 22:15:51.570794 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:51.570774 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1\": container with ID starting with 5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1 not found: ID does not exist" containerID="5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1" Nov 03 22:15:51.570853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.570803 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1"} err="failed to get container status \"5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1\": rpc error: code = NotFound desc = could not find container \"5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1\": container with ID starting with 5cf432913486c47e8c6d3c566588bf29c812f368832654169d926ceef7119df1 not found: ID does not exist" Nov 03 22:15:51.723184 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.723158 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw"] Nov 03 22:15:51.728337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:51.728319 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-c5msw"] Nov 03 22:15:52.509160 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:52.509129 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" path="/var/lib/kubelet/pods/9cb4ab36-e83f-4782-a119-b50938bcf669/volumes" Nov 03 22:15:53.133811 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:53.133781 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:53.847754 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:53.847721 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:55.481526 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.481489 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" podUID="0637122d-1b20-4e4c-9961-9d582cccf487" containerName="main" containerID="cri-o://68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7" gracePeriod=45 Nov 03 22:15:55.538885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.538846 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" podUID="0637122d-1b20-4e4c-9961-9d582cccf487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.187:8000/ready\": dial tcp 10.134.0.187:8000: connect: connection refused" Nov 03 22:15:55.754996 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.754972 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:15:55.772110 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772084 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-dshm\") pod \"0637122d-1b20-4e4c-9961-9d582cccf487\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " Nov 03 22:15:55.772219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772129 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-model-cache\") pod \"0637122d-1b20-4e4c-9961-9d582cccf487\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " Nov 03 22:15:55.772219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772191 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0637122d-1b20-4e4c-9961-9d582cccf487-kube-api-access-7b722\" (UniqueName: \"kubernetes.io/projected/0637122d-1b20-4e4c-9961-9d582cccf487-kube-api-access-7b722\") pod \"0637122d-1b20-4e4c-9961-9d582cccf487\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " Nov 03 22:15:55.772327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772226 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-home\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-home\") pod \"0637122d-1b20-4e4c-9961-9d582cccf487\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " Nov 03 22:15:55.772327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772244 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-tmp-dir\") pod \"0637122d-1b20-4e4c-9961-9d582cccf487\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " Nov 03 22:15:55.772327 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772277 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0637122d-1b20-4e4c-9961-9d582cccf487-tls-certs\" (UniqueName: \"kubernetes.io/secret/0637122d-1b20-4e4c-9961-9d582cccf487-tls-certs\") pod \"0637122d-1b20-4e4c-9961-9d582cccf487\" (UID: \"0637122d-1b20-4e4c-9961-9d582cccf487\") " Nov 03 22:15:55.772486 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772410 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-model-cache" pod "0637122d-1b20-4e4c-9961-9d582cccf487" (UID: "0637122d-1b20-4e4c-9961-9d582cccf487"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:55.772486 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772455 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-home" pod "0637122d-1b20-4e4c-9961-9d582cccf487" (UID: "0637122d-1b20-4e4c-9961-9d582cccf487"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:55.772605 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772543 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-tmp-dir" pod "0637122d-1b20-4e4c-9961-9d582cccf487" (UID: "0637122d-1b20-4e4c-9961-9d582cccf487"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:55.772978 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772953 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:55.773074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772981 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:55.773074 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.772994 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:55.774428 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.774400 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-dshm" pod "0637122d-1b20-4e4c-9961-9d582cccf487" (UID: "0637122d-1b20-4e4c-9961-9d582cccf487"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:15:55.774504 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.774482 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0637122d-1b20-4e4c-9961-9d582cccf487-tls-certs" pod "0637122d-1b20-4e4c-9961-9d582cccf487" (UID: "0637122d-1b20-4e4c-9961-9d582cccf487"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:15:55.774504 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.774483 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0637122d-1b20-4e4c-9961-9d582cccf487-kube-api-access-7b722" pod "0637122d-1b20-4e4c-9961-9d582cccf487" (UID: "0637122d-1b20-4e4c-9961-9d582cccf487"). InnerVolumeSpecName "kube-api-access-7b722". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:15:55.874068 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.874046 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b722\" (UniqueName: \"kubernetes.io/projected/0637122d-1b20-4e4c-9961-9d582cccf487-kube-api-access-7b722\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:55.874068 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.874069 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0637122d-1b20-4e4c-9961-9d582cccf487-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:55.874234 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:55.874077 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0637122d-1b20-4e4c-9961-9d582cccf487-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:15:56.586480 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.586449 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:56.595098 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.595076 5931 generic.go:356] "Generic (PLEG): container finished" podID="0637122d-1b20-4e4c-9961-9d582cccf487" containerID="68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7" exitCode=0 Nov 03 22:15:56.595241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.595112 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" Nov 03 22:15:56.595241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.595159 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" event={"ID":"0637122d-1b20-4e4c-9961-9d582cccf487","Type":"ContainerDied","Data":"68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7"} Nov 03 22:15:56.595241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.595196 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27" event={"ID":"0637122d-1b20-4e4c-9961-9d582cccf487","Type":"ContainerDied","Data":"b32d39005bef20ecdd12e33b73c9af84a834a9488823f40cb1a8bda819a0e48d"} Nov 03 22:15:56.595241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.595218 5931 scope.go:122] "RemoveContainer" containerID="68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7" Nov 03 22:15:56.604953 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.604932 5931 scope.go:122] "RemoveContainer" containerID="68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7" Nov 03 22:15:56.605236 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:15:56.605210 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7\": container with ID starting with 68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7 not found: ID does not exist" containerID="68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7" Nov 03 22:15:56.605304 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.605243 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7"} err="failed to get container status \"68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7\": rpc error: code = NotFound desc = could not find container \"68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7\": container with ID starting with 68253531dfb47e95bf0ba1b7751972323580c1bb2b8f4d94423ed8451f9038a7 not found: ID does not exist" Nov 03 22:15:56.759083 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.759059 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27"] Nov 03 22:15:56.808737 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:56.808680 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-12dee4/autores-model-12dee4-kserve-6d9c864c57-4dx27"] Nov 03 22:15:58.271899 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:58.271866 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:58.507322 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:58.507282 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0637122d-1b20-4e4c-9961-9d582cccf487" path="/var/lib/kubelet/pods/0637122d-1b20-4e4c-9961-9d582cccf487/volumes" Nov 03 22:15:58.540850 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:58.540785 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:15:58.541084 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:58.541063 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:15:59.738100 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:15:59.738055 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:00.069549 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.069461 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:00.411890 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.411862 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7676-619a7f"] Nov 03 22:16:00.412451 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.412438 5931 cpu_manager.go:414] "containerMap: removing container" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" Nov 03 22:16:00.412491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.412454 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" Nov 03 22:16:00.412528 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.412495 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0637122d-1b20-4e4c-9961-9d582cccf487" containerName="main" Nov 03 22:16:00.412528 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.412500 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0637122d-1b20-4e4c-9961-9d582cccf487" containerName="main" Nov 03 22:16:00.412601 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.412589 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0637122d-1b20-4e4c-9961-9d582cccf487" containerName="main" Nov 03 22:16:00.412660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.412602 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb4ab36-e83f-4782-a119-b50938bcf669" containerName="istio-proxy" Nov 03 22:16:00.417353 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.417336 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:00.468765 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.468742 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-619a7f"] Nov 03 22:16:00.515172 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.515148 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:16:00.515295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.515171 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8dgqp\"" Nov 03 22:16:00.515295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.515243 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:16:00.527471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.527448 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpwnf\" (UniqueName: \"kubernetes.io/projected/9583bfb5-b8a6-4eb0-945e-9c6359ae001c-kube-api-access-cpwnf\") pod \"test-curl-7676-619a7f\" (UID: \"9583bfb5-b8a6-4eb0-945e-9c6359ae001c\") " pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:00.627943 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.627919 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpwnf\" (UniqueName: \"kubernetes.io/projected/9583bfb5-b8a6-4eb0-945e-9c6359ae001c-kube-api-access-cpwnf\") pod \"test-curl-7676-619a7f\" (UID: \"9583bfb5-b8a6-4eb0-945e-9c6359ae001c\") " pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:00.684497 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.684440 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpwnf\" (UniqueName: \"kubernetes.io/projected/9583bfb5-b8a6-4eb0-945e-9c6359ae001c-kube-api-access-cpwnf\") pod \"test-curl-7676-619a7f\" (UID: \"9583bfb5-b8a6-4eb0-945e-9c6359ae001c\") " pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:00.729295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:00.729276 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:01.100323 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:01.100298 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-619a7f"] Nov 03 22:16:01.102277 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:16:01.102251 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9583bfb5_b8a6_4eb0_945e_9c6359ae001c.slice/crio-a54b927de0c5f309ee0c1947ea04f0e1676a9cc8a9ca7937338d3429f5bb31ac WatchSource:0}: Error finding container a54b927de0c5f309ee0c1947ea04f0e1676a9cc8a9ca7937338d3429f5bb31ac: Status 404 returned error can't find the container with id a54b927de0c5f309ee0c1947ea04f0e1676a9cc8a9ca7937338d3429f5bb31ac Nov 03 22:16:01.104328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:01.104313 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:16:01.629816 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:01.629787 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-619a7f" event={"ID":"9583bfb5-b8a6-4eb0-945e-9c6359ae001c","Type":"ContainerStarted","Data":"a54b927de0c5f309ee0c1947ea04f0e1676a9cc8a9ca7937338d3429f5bb31ac"} Nov 03 22:16:03.000800 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:03.000722 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:03.083129 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:16:03.083085 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:16:04.653389 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:04.653341 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-619a7f" event={"ID":"9583bfb5-b8a6-4eb0-945e-9c6359ae001c","Type":"ContainerStarted","Data":"c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613"} Nov 03 22:16:04.764764 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:04.764714 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7676-619a7f" podStartSLOduration=1.649736842 podStartE2EDuration="4.764700308s" totalImagesPullingTime="3.114963466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:04.762924535 +0000 UTC m=+3632.373772359" watchObservedRunningTime="2026-11-03 22:16:04.764700308 +0000 UTC m=+3632.375548122" Nov 03 22:16:06.291988 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:06.291958 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7676-619a7f"] Nov 03 22:16:06.295128 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:06.295103 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7676-619a7f"] Nov 03 22:16:06.674150 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:06.674061 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7676-619a7f" podUID="9583bfb5-b8a6-4eb0-945e-9c6359ae001c" containerName="test-curl-7676-619a7f" containerID="cri-o://c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613" gracePeriod=2 Nov 03 22:16:07.643482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:07.643453 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:08.072910 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:08.072844 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:08.625920 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:08.625889 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:08.866101 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:08.866073 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:08.890701 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:08.890648 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:09.013943 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.013915 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9583bfb5-b8a6-4eb0-945e-9c6359ae001c-kube-api-access-cpwnf\" (UniqueName: \"kubernetes.io/projected/9583bfb5-b8a6-4eb0-945e-9c6359ae001c-kube-api-access-cpwnf\") pod \"9583bfb5-b8a6-4eb0-945e-9c6359ae001c\" (UID: \"9583bfb5-b8a6-4eb0-945e-9c6359ae001c\") " Nov 03 22:16:09.016090 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.016044 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9583bfb5-b8a6-4eb0-945e-9c6359ae001c-kube-api-access-cpwnf" pod "9583bfb5-b8a6-4eb0-945e-9c6359ae001c" (UID: "9583bfb5-b8a6-4eb0-945e-9c6359ae001c"). InnerVolumeSpecName "kube-api-access-cpwnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:09.115084 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.115060 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpwnf\" (UniqueName: \"kubernetes.io/projected/9583bfb5-b8a6-4eb0-945e-9c6359ae001c-kube-api-access-cpwnf\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:16:09.698186 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.698156 5931 generic.go:356] "Generic (PLEG): container finished" podID="9583bfb5-b8a6-4eb0-945e-9c6359ae001c" containerID="c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613" exitCode=137 Nov 03 22:16:09.698359 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.698200 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:09.698359 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.698228 5931 scope.go:122] "RemoveContainer" containerID="c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613" Nov 03 22:16:09.708408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.708388 5931 scope.go:122] "RemoveContainer" containerID="c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613" Nov 03 22:16:09.708680 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:16:09.708660 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613\": container with ID starting with c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613 not found: ID does not exist" containerID="c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613" Nov 03 22:16:09.708747 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:09.708687 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613"} err="failed to get container status \"c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613\": rpc error: code = NotFound desc = could not find container \"c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613\": container with ID starting with c576cb49660138c772cd7448c3c7c961acd6bc9d1350c53273166daecddc9613 not found: ID does not exist" Nov 03 22:16:09.756849 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:16:09.756822 5931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7676-619a7f\" is forbidden: User \"system:node:ip-10-0-128-179.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-179.us-west-1.compute.internal' and this object" podUID="9583bfb5-b8a6-4eb0-945e-9c6359ae001c" pod="opendatahub/test-curl-7676-619a7f" Nov 03 22:16:10.507661 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:10.507606 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9583bfb5-b8a6-4eb0-945e-9c6359ae001c" path="/var/lib/kubelet/pods/9583bfb5-b8a6-4eb0-945e-9c6359ae001c/volumes" Nov 03 22:16:12.143212 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.143176 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx"] Nov 03 22:16:12.143808 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.143793 5931 cpu_manager.go:414] "containerMap: removing container" podUID="9583bfb5-b8a6-4eb0-945e-9c6359ae001c" containerName="test-curl-7676-619a7f" Nov 03 22:16:12.143882 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.143809 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9583bfb5-b8a6-4eb0-945e-9c6359ae001c" containerName="test-curl-7676-619a7f" Nov 03 22:16:12.143935 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.143926 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9583bfb5-b8a6-4eb0-945e-9c6359ae001c" containerName="test-curl-7676-619a7f" Nov 03 22:16:12.148841 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.148822 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.204641 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.204592 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx"] Nov 03 22:16:12.240265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.240239 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.240377 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.240355 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-metrics-certs\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.240428 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.240397 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.240466 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.240450 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvrl\" (UniqueName: \"kubernetes.io/projected/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-kube-api-access-jvvrl\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.249664 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.249645 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-288188-serving-cert\"" Nov 03 22:16:12.341386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.341359 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-metrics-certs\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.341528 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.341404 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.341528 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.341439 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvrl\" (UniqueName: \"kubernetes.io/projected/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-kube-api-access-jvvrl\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.341689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.341538 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.341689 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:16:12.341549 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-serving-cert: secret "maas-api-e2e-extmod-a-288188-serving-cert" not found Nov 03 22:16:12.341689 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:16:12.341601 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls podName:06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3 nodeName:}" failed. No retries permitted until 2026-11-03 22:16:12.841587795 +0000 UTC m=+3640.452435589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls") pod "maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" (UID: "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3") : secret "maas-api-e2e-extmod-a-288188-serving-cert" not found Nov 03 22:16:12.342211 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.342183 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.343843 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.343824 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-metrics-certs\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.399062 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.399002 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvrl\" (UniqueName: \"kubernetes.io/projected/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-kube-api-access-jvvrl\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.424204 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.424177 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:12.653436 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.653361 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:12.846977 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.846946 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.849366 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.849343 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:12.885094 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:12.885067 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:13.062221 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:13.062165 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:13.117936 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:13.117424 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:13.386471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:13.386436 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:13.450853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:13.450828 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx"] Nov 03 22:16:13.452659 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:16:13.452607 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d6f21b_4353_4d9c_84fd_b6d4bdfd7aa3.slice/crio-76bf993641d6e5470a4201ac83ec6201adc8e31c38d3cbff7165871eba6e3efc WatchSource:0}: Error finding container 76bf993641d6e5470a4201ac83ec6201adc8e31c38d3cbff7165871eba6e3efc: Status 404 returned error can't find the container with id 76bf993641d6e5470a4201ac83ec6201adc8e31c38d3cbff7165871eba6e3efc Nov 03 22:16:13.727222 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:13.727153 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" event={"ID":"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3","Type":"ContainerStarted","Data":"76bf993641d6e5470a4201ac83ec6201adc8e31c38d3cbff7165871eba6e3efc"} Nov 03 22:16:14.735080 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:14.735005 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" event={"ID":"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3","Type":"ContainerStarted","Data":"2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e"} Nov 03 22:16:14.735080 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:14.735062 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:14.851034 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:14.850990 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" podStartSLOduration=1.9355894980000001 podStartE2EDuration="2.850976511s" totalImagesPullingTime="915.387013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:14.848721211 +0000 UTC m=+3642.459569026" watchObservedRunningTime="2026-11-03 22:16:14.850976511 +0000 UTC m=+3642.461824326" Nov 03 22:16:16.935532 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:16.935492 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:17.200968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.200898 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:17.214576 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.214551 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw"] Nov 03 22:16:17.218880 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.218864 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.270983 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.270963 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f3e839\"" Nov 03 22:16:17.291944 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.291913 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wq6\" (UniqueName: \"kubernetes.io/projected/873d6d09-fb2f-4e52-a1e4-80e79e341abf-kube-api-access-w2wq6\") pod \"payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.292131 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.292115 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873d6d09-fb2f-4e52-a1e4-80e79e341abf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.320556 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.320518 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f3e839-dockercfg-5lrdb\"" Nov 03 22:16:17.325391 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.325363 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw"] Nov 03 22:16:17.370719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.370685 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf"] Nov 03 22:16:17.375777 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.375754 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.392941 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.392918 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873d6d09-fb2f-4e52-a1e4-80e79e341abf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.393060 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.393047 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wq6\" (UniqueName: \"kubernetes.io/projected/873d6d09-fb2f-4e52-a1e4-80e79e341abf-kube-api-access-w2wq6\") pod \"payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.393658 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.393584 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873d6d09-fb2f-4e52-a1e4-80e79e341abf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.429642 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.429591 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf"] Nov 03 22:16:17.452051 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.451992 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wq6\" (UniqueName: \"kubernetes.io/projected/873d6d09-fb2f-4e52-a1e4-80e79e341abf-kube-api-access-w2wq6\") pod \"payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.494060 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.494036 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7ec09c-f468-4401-a12d-a670b62e6570-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.494185 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.494079 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdzw\" (UniqueName: \"kubernetes.io/projected/ce7ec09c-f468-4401-a12d-a670b62e6570-kube-api-access-lpdzw\") pod \"payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.532727 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.532705 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:17.602648 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.595774 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7ec09c-f468-4401-a12d-a670b62e6570-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.602648 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.595842 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdzw\" (UniqueName: \"kubernetes.io/projected/ce7ec09c-f468-4401-a12d-a670b62e6570-kube-api-access-lpdzw\") pod \"payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.602648 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.597328 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7ec09c-f468-4401-a12d-a670b62e6570-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.654429 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.654392 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdzw\" (UniqueName: \"kubernetes.io/projected/ce7ec09c-f468-4401-a12d-a670b62e6570-kube-api-access-lpdzw\") pod \"payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.690314 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.690286 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:17.824525 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.824493 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:17.909927 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:17.909901 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw"] Nov 03 22:16:17.911899 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:16:17.911874 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873d6d09_fb2f_4e52_a1e4_80e79e341abf.slice/crio-133c2edf115a8b1782d4756aa3f06336711ffcdf02b933bf7fc0ac527adb0c54 WatchSource:0}: Error finding container 133c2edf115a8b1782d4756aa3f06336711ffcdf02b933bf7fc0ac527adb0c54: Status 404 returned error can't find the container with id 133c2edf115a8b1782d4756aa3f06336711ffcdf02b933bf7fc0ac527adb0c54 Nov 03 22:16:18.067719 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.067698 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf"] Nov 03 22:16:18.069295 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:16:18.069268 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7ec09c_f468_4401_a12d_a670b62e6570.slice/crio-a58029acb125d7aaf056d128ca75bb3d995abbe438fb64af111c8471157e15ec WatchSource:0}: Error finding container a58029acb125d7aaf056d128ca75bb3d995abbe438fb64af111c8471157e15ec: Status 404 returned error can't find the container with id a58029acb125d7aaf056d128ca75bb3d995abbe438fb64af111c8471157e15ec Nov 03 22:16:18.770738 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.770708 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" event={"ID":"ce7ec09c-f468-4401-a12d-a670b62e6570","Type":"ContainerStarted","Data":"875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7"} Nov 03 22:16:18.770984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.770756 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" event={"ID":"ce7ec09c-f468-4401-a12d-a670b62e6570","Type":"ContainerStarted","Data":"a58029acb125d7aaf056d128ca75bb3d995abbe438fb64af111c8471157e15ec"} Nov 03 22:16:18.770984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.770805 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:18.772373 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.772338 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" event={"ID":"873d6d09-fb2f-4e52-a1e4-80e79e341abf","Type":"ContainerStarted","Data":"f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef"} Nov 03 22:16:18.772477 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.772381 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" event={"ID":"873d6d09-fb2f-4e52-a1e4-80e79e341abf","Type":"ContainerStarted","Data":"133c2edf115a8b1782d4756aa3f06336711ffcdf02b933bf7fc0ac527adb0c54"} Nov 03 22:16:18.772477 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.772460 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:18.885466 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.885423 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" podStartSLOduration=1.885412794 podStartE2EDuration="1.885412794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:18.883100186 +0000 UTC m=+3646.493947998" watchObservedRunningTime="2026-11-03 22:16:18.885412794 +0000 UTC m=+3646.496260674" Nov 03 22:16:18.997186 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:18.997132 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" podStartSLOduration=1.997117427 podStartE2EDuration="1.997117427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:18.994217844 +0000 UTC m=+3646.605065661" watchObservedRunningTime="2026-11-03 22:16:18.997117427 +0000 UTC m=+3646.607965242" Nov 03 22:16:20.747230 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:20.747205 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:16:22.855663 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:22.855597 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:28.103626 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:28.103587 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:29.781468 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:29.781442 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:16:29.781901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:29.781513 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:16:33.068043 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:16:33.068005 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:16:33.139883 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:33.139855 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:38.065115 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:38.065084 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:38.305773 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:38.305734 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:38.531332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:38.531301 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:43.228189 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:43.228154 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:48.067939 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:48.067907 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:52.793122 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:52.793085 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:56.931791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:56.931743 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:57.431121 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:16:57.431091 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:02.392400 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:02.392361 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:03.068629 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:17:03.068576 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:17:06.978972 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:06.978938 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:07.234647 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:07.234556 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:11.845896 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:11.845820 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:16.642044 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:16.642011 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:21.335938 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:21.335884 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:26.020394 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:26.020351 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:29.076660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:29.076626 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:30.826980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:30.826949 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:33.068131 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:17:33.068094 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:17:35.492332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:35.492302 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:40.286657 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:40.286595 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:40.518007 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:40.517973 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:40.743578 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:40.743550 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:45.393982 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:45.393944 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:50.247581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:50.247540 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:51.398589 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:51.398559 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:55.606055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:55.606019 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:57.292020 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:57.291989 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:57.833832 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:57.833777 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:59.438881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:17:59.438848 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:00.007058 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.007024 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q"] Nov 03 22:18:00.012181 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.012158 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.063798 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.063768 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-48ae0a-openshift-default-dockercfg-fw8vc\"" Nov 03 22:18:00.093555 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093518 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-socket\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.093733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093676 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-envoy\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.093733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093715 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1ac5d-4215-4817-af62-4931f675c3db-istiod-ca-cert\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.093857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093785 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-istio-token\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.093857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093812 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1ac5d-4215-4817-af62-4931f675c3db-istio-podinfo\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.093857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093837 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ps4n\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-kube-api-access-4ps4n\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.094028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093922 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-data\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.094028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.093956 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-certs\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.094145 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.094034 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-credential-socket\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.116884 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.116847 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q"] Nov 03 22:18:00.187569 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.187505 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:00.195066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195034 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-credential-socket\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195096 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-socket\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195266 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195201 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-envoy\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195266 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195225 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1ac5d-4215-4817-af62-4931f675c3db-istiod-ca-cert\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195268 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-istio-token\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195305 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1ac5d-4215-4817-af62-4931f675c3db-istio-podinfo\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195336 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ps4n\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-kube-api-access-4ps4n\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195369 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-data\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195629 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195397 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-credential-socket\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195629 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195475 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-socket\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195629 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195475 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-certs\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195809 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195763 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-data\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195882 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195833 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-certs\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.195957 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.195937 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1ac5d-4215-4817-af62-4931f675c3db-istiod-ca-cert\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.197594 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.197569 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-envoy\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.198090 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.198074 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1ac5d-4215-4817-af62-4931f675c3db-istio-podinfo\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.251268 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.251238 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-istio-token\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.251370 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.251296 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ps4n\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-kube-api-access-4ps4n\") pod \"e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.325544 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.325488 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:00.549046 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.548992 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:00.755574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.755551 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q"] Nov 03 22:18:00.756749 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:18:00.756714 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c1ac5d_4215_4817_af62_4931f675c3db.slice/crio-5731e369c65e826132d223e955dc084a89d0a73dd549bf4d0ba03c9ff9a21a2f WatchSource:0}: Error finding container 5731e369c65e826132d223e955dc084a89d0a73dd549bf4d0ba03c9ff9a21a2f: Status 404 returned error can't find the container with id 5731e369c65e826132d223e955dc084a89d0a73dd549bf4d0ba03c9ff9a21a2f Nov 03 22:18:00.759093 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.759054 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:18:00.759188 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.759136 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:18:00.759188 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.759164 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:18:00.964237 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:00.964182 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:01.471880 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:01.471850 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" event={"ID":"30c1ac5d-4215-4817-af62-4931f675c3db","Type":"ContainerStarted","Data":"aecaacb4c87922822952a46b4c4791b451c9a2c7009728734c76dda9cd4a7d8e"} Nov 03 22:18:01.472048 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:01.471903 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" event={"ID":"30c1ac5d-4215-4817-af62-4931f675c3db","Type":"ContainerStarted","Data":"5731e369c65e826132d223e955dc084a89d0a73dd549bf4d0ba03c9ff9a21a2f"} Nov 03 22:18:01.688079 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:01.688019 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" podStartSLOduration=2.6880050840000003 podStartE2EDuration="2.688005084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:01.63582037 +0000 UTC m=+3749.246668184" watchObservedRunningTime="2026-11-03 22:18:01.688005084 +0000 UTC m=+3749.298852898" Nov 03 22:18:02.326122 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:02.326094 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:02.332708 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:02.332679 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:02.478659 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:02.478598 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:02.479358 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:02.479338 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:18:03.070270 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:18:03.070238 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:18:04.643911 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:04.643881 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:05.004201 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:05.004123 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:06.350924 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:06.350896 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:07.117760 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:07.117729 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:07.348960 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:07.348934 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:10.693330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:10.693299 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:10.918754 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:10.918722 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:11.445791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:11.445756 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:16.328071 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:16.328040 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:21.593721 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:21.593688 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:26.920331 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:26.920296 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:28.966398 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:28.966360 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8"] Nov 03 22:18:29.079336 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:29.079292 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv"] Nov 03 22:18:29.430740 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:29.430710 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:29.690963 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:29.690868 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:32.432493 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:32.432461 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:33.068928 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:18:33.068897 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:18:33.967318 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:33.967281 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" podUID="3b27ab41-5187-4091-b628-b56d17fb8658" containerName="payload-processing" containerID="cri-o://66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30" gracePeriod=25 Nov 03 22:18:34.079951 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.079897 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" podUID="c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" containerName="payload-pre-processing" containerID="cri-o://b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6" gracePeriod=25 Nov 03 22:18:34.243559 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.243535 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:18:34.324452 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.324418 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b27ab41-5187-4091-b628-b56d17fb8658-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b27ab41-5187-4091-b628-b56d17fb8658-plugins-config-volume\") pod \"3b27ab41-5187-4091-b628-b56d17fb8658\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " Nov 03 22:18:34.324780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.324760 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b27ab41-5187-4091-b628-b56d17fb8658-kube-api-access-fw5ls\" (UniqueName: \"kubernetes.io/projected/3b27ab41-5187-4091-b628-b56d17fb8658-kube-api-access-fw5ls\") pod \"3b27ab41-5187-4091-b628-b56d17fb8658\" (UID: \"3b27ab41-5187-4091-b628-b56d17fb8658\") " Nov 03 22:18:34.326061 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.326027 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b27ab41-5187-4091-b628-b56d17fb8658-plugins-config-volume" pod "3b27ab41-5187-4091-b628-b56d17fb8658" (UID: "3b27ab41-5187-4091-b628-b56d17fb8658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:18:34.336885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.336828 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b27ab41-5187-4091-b628-b56d17fb8658-kube-api-access-fw5ls" pod "3b27ab41-5187-4091-b628-b56d17fb8658" (UID: "3b27ab41-5187-4091-b628-b56d17fb8658"). InnerVolumeSpecName "kube-api-access-fw5ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:34.360865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.360834 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:18:34.426293 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.426265 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-kube-api-access-8ddmq\" (UniqueName: \"kubernetes.io/projected/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-kube-api-access-8ddmq\") pod \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " Nov 03 22:18:34.426464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.426298 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-plugins-config-volume\") pod \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\" (UID: \"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10\") " Nov 03 22:18:34.426724 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.426701 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-plugins-config-volume" pod "c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" (UID: "c0f5ffad-a1ef-4381-b6e9-4d133b50ad10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:18:34.426863 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.426844 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:34.426933 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.426870 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b27ab41-5187-4091-b628-b56d17fb8658-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:34.426933 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.426881 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5ls\" (UniqueName: \"kubernetes.io/projected/3b27ab41-5187-4091-b628-b56d17fb8658-kube-api-access-fw5ls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:34.428532 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.428512 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-kube-api-access-8ddmq" pod "c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" (UID: "c0f5ffad-a1ef-4381-b6e9-4d133b50ad10"). InnerVolumeSpecName "kube-api-access-8ddmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:34.528043 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.528019 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ddmq\" (UniqueName: \"kubernetes.io/projected/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10-kube-api-access-8ddmq\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:34.701813 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.701784 5931 generic.go:356] "Generic (PLEG): container finished" podID="c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" containerID="b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6" exitCode=0 Nov 03 22:18:34.701929 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.701825 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" Nov 03 22:18:34.701929 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.701871 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" event={"ID":"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10","Type":"ContainerDied","Data":"b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6"} Nov 03 22:18:34.701929 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.701915 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv" event={"ID":"c0f5ffad-a1ef-4381-b6e9-4d133b50ad10","Type":"ContainerDied","Data":"e84e55d62d0fcc457a5d7487721c3e369be7ac98902ff0de53dab8ee761d7d70"} Nov 03 22:18:34.702069 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.701937 5931 scope.go:122] "RemoveContainer" containerID="b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6" Nov 03 22:18:34.703449 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.703427 5931 generic.go:356] "Generic (PLEG): container finished" podID="3b27ab41-5187-4091-b628-b56d17fb8658" containerID="66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30" exitCode=0 Nov 03 22:18:34.703563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.703469 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" Nov 03 22:18:34.703563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.703492 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" event={"ID":"3b27ab41-5187-4091-b628-b56d17fb8658","Type":"ContainerDied","Data":"66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30"} Nov 03 22:18:34.703563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.703521 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8" event={"ID":"3b27ab41-5187-4091-b628-b56d17fb8658","Type":"ContainerDied","Data":"891cf5949f562d99ec1d57b8d380a8da318188f781f22a58c0ce4bc88dc6ef67"} Nov 03 22:18:34.712128 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.712102 5931 scope.go:122] "RemoveContainer" containerID="b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6" Nov 03 22:18:34.712434 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:18:34.712402 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6\": container with ID starting with b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6 not found: ID does not exist" containerID="b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6" Nov 03 22:18:34.712515 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.712446 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6"} err="failed to get container status \"b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6\": rpc error: code = NotFound desc = could not find container \"b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6\": container with ID starting with b32e07796c9fd9fc21f44ba7b00ab09cd4a4cc484ddc0f920b5cc804db95c9a6 not found: ID does not exist" Nov 03 22:18:34.712515 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.712464 5931 scope.go:122] "RemoveContainer" containerID="66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30" Nov 03 22:18:34.721720 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.721699 5931 scope.go:122] "RemoveContainer" containerID="66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30" Nov 03 22:18:34.721958 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:18:34.721941 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30\": container with ID starting with 66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30 not found: ID does not exist" containerID="66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30" Nov 03 22:18:34.722012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.721965 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30"} err="failed to get container status \"66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30\": rpc error: code = NotFound desc = could not find container \"66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30\": container with ID starting with 66eec6eaeff6f56c062f9c92aff917bd8e10ef55ea51d89e6943a4eadbb3cf30 not found: ID does not exist" Nov 03 22:18:34.865695 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.865669 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv"] Nov 03 22:18:34.914883 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.914862 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5b1489-67bd9744f5-884zv"] Nov 03 22:18:34.983649 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.983360 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8"] Nov 03 22:18:34.988346 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:34.988329 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5b1489-568b5fb696-slhf8"] Nov 03 22:18:36.506048 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:36.506019 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b27ab41-5187-4091-b628-b56d17fb8658" path="/var/lib/kubelet/pods/3b27ab41-5187-4091-b628-b56d17fb8658/volumes" Nov 03 22:18:36.506435 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:36.506394 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" path="/var/lib/kubelet/pods/c0f5ffad-a1ef-4381-b6e9-4d133b50ad10/volumes" Nov 03 22:18:37.667663 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:37.667630 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:42.674720 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:42.674624 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:47.821494 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:47.821464 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:48.227852 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:48.227819 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r"] Nov 03 22:18:48.335770 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:48.335740 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q"] Nov 03 22:18:48.688901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:48.688869 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:48.915206 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:48.915178 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:52.838106 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:52.838067 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:53.228192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.228160 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" podUID="392dfab9-5b10-4ea9-b752-2dbedecfabea" containerName="payload-processing" containerID="cri-o://dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72" gracePeriod=25 Nov 03 22:18:53.336351 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.336323 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" podUID="8a523e83-dfb0-42a8-ac6a-99d640044275" containerName="payload-pre-processing" containerID="cri-o://095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71" gracePeriod=25 Nov 03 22:18:53.508164 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.508134 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:18:53.603470 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.603448 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:18:53.614884 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.614859 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392dfab9-5b10-4ea9-b752-2dbedecfabea-kube-api-access-r9n86\" (UniqueName: \"kubernetes.io/projected/392dfab9-5b10-4ea9-b752-2dbedecfabea-kube-api-access-r9n86\") pod \"392dfab9-5b10-4ea9-b752-2dbedecfabea\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " Nov 03 22:18:53.615042 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.615030 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392dfab9-5b10-4ea9-b752-2dbedecfabea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392dfab9-5b10-4ea9-b752-2dbedecfabea-plugins-config-volume\") pod \"392dfab9-5b10-4ea9-b752-2dbedecfabea\" (UID: \"392dfab9-5b10-4ea9-b752-2dbedecfabea\") " Nov 03 22:18:53.615369 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.615349 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392dfab9-5b10-4ea9-b752-2dbedecfabea-plugins-config-volume" pod "392dfab9-5b10-4ea9-b752-2dbedecfabea" (UID: "392dfab9-5b10-4ea9-b752-2dbedecfabea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:18:53.616934 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.616914 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392dfab9-5b10-4ea9-b752-2dbedecfabea-kube-api-access-r9n86" pod "392dfab9-5b10-4ea9-b752-2dbedecfabea" (UID: "392dfab9-5b10-4ea9-b752-2dbedecfabea"). InnerVolumeSpecName "kube-api-access-r9n86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:53.715937 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.715898 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a523e83-dfb0-42a8-ac6a-99d640044275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a523e83-dfb0-42a8-ac6a-99d640044275-plugins-config-volume\") pod \"8a523e83-dfb0-42a8-ac6a-99d640044275\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " Nov 03 22:18:53.716102 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.716060 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a523e83-dfb0-42a8-ac6a-99d640044275-kube-api-access-jw2jv\" (UniqueName: \"kubernetes.io/projected/8a523e83-dfb0-42a8-ac6a-99d640044275-kube-api-access-jw2jv\") pod \"8a523e83-dfb0-42a8-ac6a-99d640044275\" (UID: \"8a523e83-dfb0-42a8-ac6a-99d640044275\") " Nov 03 22:18:53.716245 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.716225 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a523e83-dfb0-42a8-ac6a-99d640044275-plugins-config-volume" pod "8a523e83-dfb0-42a8-ac6a-99d640044275" (UID: "8a523e83-dfb0-42a8-ac6a-99d640044275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:18:53.716513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.716493 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a523e83-dfb0-42a8-ac6a-99d640044275-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:53.716555 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.716520 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392dfab9-5b10-4ea9-b752-2dbedecfabea-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:53.716555 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.716535 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9n86\" (UniqueName: \"kubernetes.io/projected/392dfab9-5b10-4ea9-b752-2dbedecfabea-kube-api-access-r9n86\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:53.718117 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.718096 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a523e83-dfb0-42a8-ac6a-99d640044275-kube-api-access-jw2jv" pod "8a523e83-dfb0-42a8-ac6a-99d640044275" (UID: "8a523e83-dfb0-42a8-ac6a-99d640044275"). InnerVolumeSpecName "kube-api-access-jw2jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:53.817248 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.817184 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw2jv\" (UniqueName: \"kubernetes.io/projected/8a523e83-dfb0-42a8-ac6a-99d640044275-kube-api-access-jw2jv\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:18:53.833144 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.833113 5931 generic.go:356] "Generic (PLEG): container finished" podID="8a523e83-dfb0-42a8-ac6a-99d640044275" containerID="095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71" exitCode=0 Nov 03 22:18:53.833258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.833155 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" event={"ID":"8a523e83-dfb0-42a8-ac6a-99d640044275","Type":"ContainerDied","Data":"095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71"} Nov 03 22:18:53.833258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.833173 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" Nov 03 22:18:53.833258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.833238 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q" event={"ID":"8a523e83-dfb0-42a8-ac6a-99d640044275","Type":"ContainerDied","Data":"95d35c64b14a195d7efaba80847b59db6359367010a08ec81ab539031a5d0bc0"} Nov 03 22:18:53.833378 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.833264 5931 scope.go:122] "RemoveContainer" containerID="095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71" Nov 03 22:18:53.834742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.834720 5931 generic.go:356] "Generic (PLEG): container finished" podID="392dfab9-5b10-4ea9-b752-2dbedecfabea" containerID="dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72" exitCode=0 Nov 03 22:18:53.835012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.834780 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" Nov 03 22:18:53.835012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.834830 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" event={"ID":"392dfab9-5b10-4ea9-b752-2dbedecfabea","Type":"ContainerDied","Data":"dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72"} Nov 03 22:18:53.835012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.834882 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r" event={"ID":"392dfab9-5b10-4ea9-b752-2dbedecfabea","Type":"ContainerDied","Data":"5980f4a787486329979d1d279a039d1d54127cc854712437132e2d64dbe1b78f"} Nov 03 22:18:53.844838 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.844818 5931 scope.go:122] "RemoveContainer" containerID="095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71" Nov 03 22:18:53.845143 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:18:53.845097 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71\": container with ID starting with 095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71 not found: ID does not exist" containerID="095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71" Nov 03 22:18:53.845143 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.845120 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71"} err="failed to get container status \"095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71\": rpc error: code = NotFound desc = could not find container \"095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71\": container with ID starting with 095337a314959e51bd09a62fef0a1438bef3d1dbc0e762f668f998be09dd5a71 not found: ID does not exist" Nov 03 22:18:53.845143 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.845136 5931 scope.go:122] "RemoveContainer" containerID="dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72" Nov 03 22:18:53.854448 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.854433 5931 scope.go:122] "RemoveContainer" containerID="dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72" Nov 03 22:18:53.854711 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:18:53.854695 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72\": container with ID starting with dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72 not found: ID does not exist" containerID="dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72" Nov 03 22:18:53.854772 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.854717 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72"} err="failed to get container status \"dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72\": rpc error: code = NotFound desc = could not find container \"dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72\": container with ID starting with dc75ba4fcb71acb341f33bbd27fd9dc4cabdc4736138b8a2bbf9cfb1cc6f8a72 not found: ID does not exist" Nov 03 22:18:53.995539 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:53.995514 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q"] Nov 03 22:18:54.000888 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:54.000867 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d7b12e-6b5fd7544c-h5f8q"] Nov 03 22:18:54.108083 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:54.108055 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r"] Nov 03 22:18:54.112148 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:54.112120 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d7b12e-749f4789b6-nwq6r"] Nov 03 22:18:54.506408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:54.506350 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392dfab9-5b10-4ea9-b752-2dbedecfabea" path="/var/lib/kubelet/pods/392dfab9-5b10-4ea9-b752-2dbedecfabea/volumes" Nov 03 22:18:54.506728 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:54.506714 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a523e83-dfb0-42a8-ac6a-99d640044275" path="/var/lib/kubelet/pods/8a523e83-dfb0-42a8-ac6a-99d640044275/volumes" Nov 03 22:18:57.685334 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:57.685300 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:57.934503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:18:57.934476 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:02.345388 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:02.345353 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:03.070217 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:03.070180 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:19:07.036398 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:07.036365 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:11.822470 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:11.822433 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:16.864791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:16.864761 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:21.084858 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.084827 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx"] Nov 03 22:19:21.085254 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.085022 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" podUID="06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" containerName="maas-api" containerID="cri-o://2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e" gracePeriod=30 Nov 03 22:19:21.283678 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.283641 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:21.340778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.340756 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:19:21.381947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.381922 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-openshift-service-ca\") pod \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " Nov 03 22:19:21.382064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.382042 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-metrics-certs\") pod \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " Nov 03 22:19:21.382225 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.382205 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-kube-api-access-jvvrl\" (UniqueName: \"kubernetes.io/projected/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-kube-api-access-jvvrl\") pod \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " Nov 03 22:19:21.382279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.382250 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls\") pod \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\" (UID: \"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3\") " Nov 03 22:19:21.382279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.382262 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-openshift-service-ca" pod "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" (UID: "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:19:21.382684 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.382664 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:21.384231 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.384209 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-metrics-certs" pod "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" (UID: "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:19:21.384313 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.384266 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls" pod "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" (UID: "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:19:21.384313 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.384274 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-kube-api-access-jvvrl" pod "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" (UID: "06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3"). InnerVolumeSpecName "kube-api-access-jvvrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:21.483567 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.483543 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvvrl\" (UniqueName: \"kubernetes.io/projected/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-kube-api-access-jvvrl\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:21.483567 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.483569 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:21.483734 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:21.483579 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:22.017112 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.017079 5931 generic.go:356] "Generic (PLEG): container finished" podID="06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" containerID="2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e" exitCode=0 Nov 03 22:19:22.017295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.017125 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" Nov 03 22:19:22.017295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.017146 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" event={"ID":"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3","Type":"ContainerDied","Data":"2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e"} Nov 03 22:19:22.017295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.017195 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx" event={"ID":"06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3","Type":"ContainerDied","Data":"76bf993641d6e5470a4201ac83ec6201adc8e31c38d3cbff7165871eba6e3efc"} Nov 03 22:19:22.017295 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.017225 5931 scope.go:122] "RemoveContainer" containerID="2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e" Nov 03 22:19:22.028987 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.028962 5931 scope.go:122] "RemoveContainer" containerID="2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e" Nov 03 22:19:22.029280 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:22.029248 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e\": container with ID starting with 2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e not found: ID does not exist" containerID="2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e" Nov 03 22:19:22.029353 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.029286 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e"} err="failed to get container status \"2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e\": rpc error: code = NotFound desc = could not find container \"2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e\": container with ID starting with 2a20d6d9bbd46af4a18adb65e54cceab456ee1e8528e29053a93c7de2d82ec2e not found: ID does not exist" Nov 03 22:19:22.180030 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.179997 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx"] Nov 03 22:19:22.182883 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.182859 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-288188-8589c7c5f4-khxfx"] Nov 03 22:19:22.421070 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.421041 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf"] Nov 03 22:19:22.506798 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.506764 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" path="/var/lib/kubelet/pods/06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3/volumes" Nov 03 22:19:22.526181 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.526161 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw"] Nov 03 22:19:22.883103 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:22.883065 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:23.140903 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:23.140789 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:23.363724 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:23.363690 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:23.584842 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:23.584752 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:23.925264 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:23.925208 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:27.421352 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.421318 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" podUID="ce7ec09c-f468-4401-a12d-a670b62e6570" containerName="payload-processing" containerID="cri-o://875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7" gracePeriod=25 Nov 03 22:19:27.526597 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.526563 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" podUID="873d6d09-fb2f-4e52-a1e4-80e79e341abf" containerName="payload-pre-processing" containerID="cri-o://f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef" gracePeriod=25 Nov 03 22:19:27.709707 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.709649 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:19:27.752200 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.752170 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7ec09c-f468-4401-a12d-a670b62e6570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7ec09c-f468-4401-a12d-a670b62e6570-plugins-config-volume\") pod \"ce7ec09c-f468-4401-a12d-a670b62e6570\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " Nov 03 22:19:27.752420 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.752395 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7ec09c-f468-4401-a12d-a670b62e6570-kube-api-access-lpdzw\" (UniqueName: \"kubernetes.io/projected/ce7ec09c-f468-4401-a12d-a670b62e6570-kube-api-access-lpdzw\") pod \"ce7ec09c-f468-4401-a12d-a670b62e6570\" (UID: \"ce7ec09c-f468-4401-a12d-a670b62e6570\") " Nov 03 22:19:27.752571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.752545 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7ec09c-f468-4401-a12d-a670b62e6570-plugins-config-volume" pod "ce7ec09c-f468-4401-a12d-a670b62e6570" (UID: "ce7ec09c-f468-4401-a12d-a670b62e6570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:19:27.753048 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.752919 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7ec09c-f468-4401-a12d-a670b62e6570-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:27.758294 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.758272 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7ec09c-f468-4401-a12d-a670b62e6570-kube-api-access-lpdzw" pod "ce7ec09c-f468-4401-a12d-a670b62e6570" (UID: "ce7ec09c-f468-4401-a12d-a670b62e6570"). InnerVolumeSpecName "kube-api-access-lpdzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:27.806070 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.806046 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:19:27.824883 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.824857 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:27.854081 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.854051 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873d6d09-fb2f-4e52-a1e4-80e79e341abf-kube-api-access-w2wq6\" (UniqueName: \"kubernetes.io/projected/873d6d09-fb2f-4e52-a1e4-80e79e341abf-kube-api-access-w2wq6\") pod \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " Nov 03 22:19:27.854236 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.854111 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873d6d09-fb2f-4e52-a1e4-80e79e341abf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873d6d09-fb2f-4e52-a1e4-80e79e341abf-plugins-config-volume\") pod \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\" (UID: \"873d6d09-fb2f-4e52-a1e4-80e79e341abf\") " Nov 03 22:19:27.854579 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.854538 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873d6d09-fb2f-4e52-a1e4-80e79e341abf-plugins-config-volume" pod "873d6d09-fb2f-4e52-a1e4-80e79e341abf" (UID: "873d6d09-fb2f-4e52-a1e4-80e79e341abf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:19:27.854888 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.854865 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpdzw\" (UniqueName: \"kubernetes.io/projected/ce7ec09c-f468-4401-a12d-a670b62e6570-kube-api-access-lpdzw\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:27.855128 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.854889 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873d6d09-fb2f-4e52-a1e4-80e79e341abf-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:27.857647 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.857604 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873d6d09-fb2f-4e52-a1e4-80e79e341abf-kube-api-access-w2wq6" pod "873d6d09-fb2f-4e52-a1e4-80e79e341abf" (UID: "873d6d09-fb2f-4e52-a1e4-80e79e341abf"). InnerVolumeSpecName "kube-api-access-w2wq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:27.956350 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:27.956285 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wq6\" (UniqueName: \"kubernetes.io/projected/873d6d09-fb2f-4e52-a1e4-80e79e341abf-kube-api-access-w2wq6\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:28.062236 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.062200 5931 generic.go:356] "Generic (PLEG): container finished" podID="ce7ec09c-f468-4401-a12d-a670b62e6570" containerID="875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7" exitCode=0 Nov 03 22:19:28.062434 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.062255 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" Nov 03 22:19:28.062434 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.062289 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" event={"ID":"ce7ec09c-f468-4401-a12d-a670b62e6570","Type":"ContainerDied","Data":"875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7"} Nov 03 22:19:28.062434 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.062344 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf" event={"ID":"ce7ec09c-f468-4401-a12d-a670b62e6570","Type":"ContainerDied","Data":"a58029acb125d7aaf056d128ca75bb3d995abbe438fb64af111c8471157e15ec"} Nov 03 22:19:28.062434 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.062366 5931 scope.go:122] "RemoveContainer" containerID="875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7" Nov 03 22:19:28.063744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.063721 5931 generic.go:356] "Generic (PLEG): container finished" podID="873d6d09-fb2f-4e52-a1e4-80e79e341abf" containerID="f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef" exitCode=0 Nov 03 22:19:28.063886 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.063829 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" Nov 03 22:19:28.063886 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.063849 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" event={"ID":"873d6d09-fb2f-4e52-a1e4-80e79e341abf","Type":"ContainerDied","Data":"f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef"} Nov 03 22:19:28.064413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.063890 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw" event={"ID":"873d6d09-fb2f-4e52-a1e4-80e79e341abf","Type":"ContainerDied","Data":"133c2edf115a8b1782d4756aa3f06336711ffcdf02b933bf7fc0ac527adb0c54"} Nov 03 22:19:28.072672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.072656 5931 scope.go:122] "RemoveContainer" containerID="875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7" Nov 03 22:19:28.072938 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:28.072909 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7\": container with ID starting with 875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7 not found: ID does not exist" containerID="875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7" Nov 03 22:19:28.073028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.072954 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7"} err="failed to get container status \"875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7\": rpc error: code = NotFound desc = could not find container \"875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7\": container with ID starting with 875577dcc9a59f2056078c36b264ea3e16c8f2794113273f22a37900f84774a7 not found: ID does not exist" Nov 03 22:19:28.073028 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.072969 5931 scope.go:122] "RemoveContainer" containerID="f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef" Nov 03 22:19:28.082558 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.082542 5931 scope.go:122] "RemoveContainer" containerID="f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef" Nov 03 22:19:28.082875 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:28.082857 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef\": container with ID starting with f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef not found: ID does not exist" containerID="f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef" Nov 03 22:19:28.082951 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.082885 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef"} err="failed to get container status \"f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef\": rpc error: code = NotFound desc = could not find container \"f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef\": container with ID starting with f949ce249f341fb2b01fdf2e02f9995b3561851e49878fc47f370e7af1f9ddef not found: ID does not exist" Nov 03 22:19:28.229603 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.229531 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf"] Nov 03 22:19:28.233009 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.232989 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f3e839-7d9864b4-lq5nf"] Nov 03 22:19:28.340258 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.340233 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw"] Nov 03 22:19:28.346206 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.346185 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f3e839-5dd976475f-zjppw"] Nov 03 22:19:28.506484 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.506412 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873d6d09-fb2f-4e52-a1e4-80e79e341abf" path="/var/lib/kubelet/pods/873d6d09-fb2f-4e52-a1e4-80e79e341abf/volumes" Nov 03 22:19:28.506976 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:28.506955 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7ec09c-f468-4401-a12d-a670b62e6570" path="/var/lib/kubelet/pods/ce7ec09c-f468-4401-a12d-a670b62e6570/volumes" Nov 03 22:19:29.322773 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:29.322736 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:29.552754 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:29.552726 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:29.846495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:29.846462 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:30.242717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:30.242634 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:30.467174 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:30.467149 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:33.068452 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:33.068412 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:19:34.235014 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.234981 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz"] Nov 03 22:19:34.235754 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235736 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" containerName="payload-pre-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235756 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" containerName="payload-pre-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235795 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8a523e83-dfb0-42a8-ac6a-99d640044275" containerName="payload-pre-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235805 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a523e83-dfb0-42a8-ac6a-99d640044275" containerName="payload-pre-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235817 5931 cpu_manager.go:414] "containerMap: removing container" podUID="873d6d09-fb2f-4e52-a1e4-80e79e341abf" containerName="payload-pre-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235823 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873d6d09-fb2f-4e52-a1e4-80e79e341abf" containerName="payload-pre-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235841 5931 cpu_manager.go:414] "containerMap: removing container" podUID="392dfab9-5b10-4ea9-b752-2dbedecfabea" containerName="payload-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235846 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392dfab9-5b10-4ea9-b752-2dbedecfabea" containerName="payload-processing" Nov 03 22:19:34.235865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235868 5931 cpu_manager.go:414] "containerMap: removing container" podUID="06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" containerName="maas-api" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235876 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" containerName="maas-api" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235891 5931 cpu_manager.go:414] "containerMap: removing container" podUID="ce7ec09c-f468-4401-a12d-a670b62e6570" containerName="payload-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235897 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7ec09c-f468-4401-a12d-a670b62e6570" containerName="payload-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235911 5931 cpu_manager.go:414] "containerMap: removing container" podUID="3b27ab41-5187-4091-b628-b56d17fb8658" containerName="payload-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.235917 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b27ab41-5187-4091-b628-b56d17fb8658" containerName="payload-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.236043 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="392dfab9-5b10-4ea9-b752-2dbedecfabea" containerName="payload-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.236060 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d6f21b-4353-4d9c-84fd-b6d4bdfd7aa3" containerName="maas-api" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.236072 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7ec09c-f468-4401-a12d-a670b62e6570" containerName="payload-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.236081 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b27ab41-5187-4091-b628-b56d17fb8658" containerName="payload-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.236094 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f5ffad-a1ef-4381-b6e9-4d133b50ad10" containerName="payload-pre-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.236107 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a523e83-dfb0-42a8-ac6a-99d640044275" containerName="payload-pre-processing" Nov 03 22:19:34.236166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.236119 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="873d6d09-fb2f-4e52-a1e4-80e79e341abf" containerName="payload-pre-processing" Nov 03 22:19:34.241622 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.241589 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.295751 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.295724 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz"] Nov 03 22:19:34.314302 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.314276 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.314404 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.314331 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/479afc9c-4f61-40bb-8b23-5de937c635c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.314462 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.314443 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx54s\" (UniqueName: \"kubernetes.io/projected/479afc9c-4f61-40bb-8b23-5de937c635c3-kube-api-access-xx54s\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.314571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.314554 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.341061 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.341042 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-48ae0a-serving-cert\"" Nov 03 22:19:34.415938 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.415912 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/479afc9c-4f61-40bb-8b23-5de937c635c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.416046 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.415998 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx54s\" (UniqueName: \"kubernetes.io/projected/479afc9c-4f61-40bb-8b23-5de937c635c3-kube-api-access-xx54s\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.416119 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.416078 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.416119 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.416099 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.416237 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:34.416219 5931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-serving-cert: secret "maas-api-e2e-extmod-a-48ae0a-serving-cert" not found Nov 03 22:19:34.416310 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:34.416294 5931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls podName:479afc9c-4f61-40bb-8b23-5de937c635c3 nodeName:}" failed. No retries permitted until 2026-11-03 22:19:34.916280081 +0000 UTC m=+3842.527127874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls") pod "maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" (UID: "479afc9c-4f61-40bb-8b23-5de937c635c3") : secret "maas-api-e2e-extmod-a-48ae0a-serving-cert" not found Nov 03 22:19:34.416657 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.416598 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/479afc9c-4f61-40bb-8b23-5de937c635c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.418742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.418723 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.472973 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.472953 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx54s\" (UniqueName: \"kubernetes.io/projected/479afc9c-4f61-40bb-8b23-5de937c635c3-kube-api-access-xx54s\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.531535 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.531454 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:34.762539 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.762506 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:34.921940 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.921912 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:34.924427 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:34.924399 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:35.155851 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:35.155830 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:35.398253 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:35.398228 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:35.530838 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:35.530813 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz"] Nov 03 22:19:35.533587 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:19:35.533562 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479afc9c_4f61_40bb_8b23_5de937c635c3.slice/crio-c34c5f95fd8a705a6b478c505bc1c8451f6b1cdf26ee85442e4786191b8b7fe3 WatchSource:0}: Error finding container c34c5f95fd8a705a6b478c505bc1c8451f6b1cdf26ee85442e4786191b8b7fe3: Status 404 returned error can't find the container with id c34c5f95fd8a705a6b478c505bc1c8451f6b1cdf26ee85442e4786191b8b7fe3 Nov 03 22:19:35.632916 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:35.632877 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:35.908410 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:35.908384 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:36.119585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:36.119546 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" event={"ID":"479afc9c-4f61-40bb-8b23-5de937c635c3","Type":"ContainerStarted","Data":"c34c5f95fd8a705a6b478c505bc1c8451f6b1cdf26ee85442e4786191b8b7fe3"} Nov 03 22:19:36.140824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:36.140784 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:36.389064 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:36.389035 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:37.127785 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:37.127746 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" event={"ID":"479afc9c-4f61-40bb-8b23-5de937c635c3","Type":"ContainerStarted","Data":"e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1"} Nov 03 22:19:37.128187 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:37.127813 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:37.242662 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:37.242600 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" podStartSLOduration=2.428864276 podStartE2EDuration="3.242587074s" totalImagesPullingTime="813.722798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:37.240504667 +0000 UTC m=+3844.851352481" watchObservedRunningTime="2026-11-03 22:19:37.242587074 +0000 UTC m=+3844.853434888" Nov 03 22:19:37.608442 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:37.608407 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:37.831954 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:37.831919 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:39.876852 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:39.876812 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:43.139456 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:43.139429 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:19:43.732099 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:43.732071 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw"] Nov 03 22:19:43.839651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:43.839600 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z"] Nov 03 22:19:44.190804 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:44.190773 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:44.417894 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:44.417861 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:45.064013 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:45.063981 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:48.497381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:48.497348 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk"] Nov 03 22:19:48.606665 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:48.606633 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9"] Nov 03 22:19:48.733188 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:48.733159 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" podUID="85227cfc-1db8-46e0-964a-c24442389ad2" containerName="payload-processing" containerID="cri-o://5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2" gracePeriod=25 Nov 03 22:19:48.840409 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:48.840353 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" podUID="15288224-fbef-47db-b0a5-8d4f488a6915" containerName="payload-pre-processing" containerID="cri-o://c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2" gracePeriod=25 Nov 03 22:19:48.955406 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:48.955324 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:49.005397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.005343 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:19:49.064928 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.064902 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85227cfc-1db8-46e0-964a-c24442389ad2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85227cfc-1db8-46e0-964a-c24442389ad2-plugins-config-volume\") pod \"85227cfc-1db8-46e0-964a-c24442389ad2\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " Nov 03 22:19:49.065108 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.064947 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85227cfc-1db8-46e0-964a-c24442389ad2-kube-api-access-d6q66\" (UniqueName: \"kubernetes.io/projected/85227cfc-1db8-46e0-964a-c24442389ad2-kube-api-access-d6q66\") pod \"85227cfc-1db8-46e0-964a-c24442389ad2\" (UID: \"85227cfc-1db8-46e0-964a-c24442389ad2\") " Nov 03 22:19:49.065311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.065287 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85227cfc-1db8-46e0-964a-c24442389ad2-plugins-config-volume" pod "85227cfc-1db8-46e0-964a-c24442389ad2" (UID: "85227cfc-1db8-46e0-964a-c24442389ad2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:19:49.067753 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.067730 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85227cfc-1db8-46e0-964a-c24442389ad2-kube-api-access-d6q66" pod "85227cfc-1db8-46e0-964a-c24442389ad2" (UID: "85227cfc-1db8-46e0-964a-c24442389ad2"). InnerVolumeSpecName "kube-api-access-d6q66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:49.133415 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.133392 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:19:49.166014 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.165992 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15288224-fbef-47db-b0a5-8d4f488a6915-kube-api-access-m5psk\" (UniqueName: \"kubernetes.io/projected/15288224-fbef-47db-b0a5-8d4f488a6915-kube-api-access-m5psk\") pod \"15288224-fbef-47db-b0a5-8d4f488a6915\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " Nov 03 22:19:49.166118 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.166078 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15288224-fbef-47db-b0a5-8d4f488a6915-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15288224-fbef-47db-b0a5-8d4f488a6915-plugins-config-volume\") pod \"15288224-fbef-47db-b0a5-8d4f488a6915\" (UID: \"15288224-fbef-47db-b0a5-8d4f488a6915\") " Nov 03 22:19:49.166479 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.166449 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15288224-fbef-47db-b0a5-8d4f488a6915-plugins-config-volume" pod "15288224-fbef-47db-b0a5-8d4f488a6915" (UID: "15288224-fbef-47db-b0a5-8d4f488a6915"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:19:49.166571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.166561 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85227cfc-1db8-46e0-964a-c24442389ad2-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:49.166637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.166576 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6q66\" (UniqueName: \"kubernetes.io/projected/85227cfc-1db8-46e0-964a-c24442389ad2-kube-api-access-d6q66\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:49.166637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.166585 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15288224-fbef-47db-b0a5-8d4f488a6915-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:49.168046 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.168028 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15288224-fbef-47db-b0a5-8d4f488a6915-kube-api-access-m5psk" pod "15288224-fbef-47db-b0a5-8d4f488a6915" (UID: "15288224-fbef-47db-b0a5-8d4f488a6915"). InnerVolumeSpecName "kube-api-access-m5psk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:49.200879 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.200856 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:49.209812 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.209787 5931 generic.go:356] "Generic (PLEG): container finished" podID="15288224-fbef-47db-b0a5-8d4f488a6915" containerID="c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2" exitCode=0 Nov 03 22:19:49.209911 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.209816 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" event={"ID":"15288224-fbef-47db-b0a5-8d4f488a6915","Type":"ContainerDied","Data":"c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2"} Nov 03 22:19:49.209911 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.209836 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" Nov 03 22:19:49.209911 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.209885 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z" event={"ID":"15288224-fbef-47db-b0a5-8d4f488a6915","Type":"ContainerDied","Data":"ec97caa5a80acad381cd97f076f1ad8bddaa49260b7c64f06cf0c734b78e333a"} Nov 03 22:19:49.209911 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.209910 5931 scope.go:122] "RemoveContainer" containerID="c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2" Nov 03 22:19:49.211414 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.211392 5931 generic.go:356] "Generic (PLEG): container finished" podID="85227cfc-1db8-46e0-964a-c24442389ad2" containerID="5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2" exitCode=0 Nov 03 22:19:49.211537 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.211437 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" event={"ID":"85227cfc-1db8-46e0-964a-c24442389ad2","Type":"ContainerDied","Data":"5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2"} Nov 03 22:19:49.211537 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.211432 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" Nov 03 22:19:49.211537 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.211476 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw" event={"ID":"85227cfc-1db8-46e0-964a-c24442389ad2","Type":"ContainerDied","Data":"3b65b5ead7b9116bcc3b8004af28eb3000824f24bf974a98ab2cbb176b112bfc"} Nov 03 22:19:49.221397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.221378 5931 scope.go:122] "RemoveContainer" containerID="c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2" Nov 03 22:19:49.221694 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:49.221674 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2\": container with ID starting with c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2 not found: ID does not exist" containerID="c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2" Nov 03 22:19:49.221742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.221702 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2"} err="failed to get container status \"c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2\": rpc error: code = NotFound desc = could not find container \"c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2\": container with ID starting with c33dd489fed4085c65fb30f8bb60af17f2b32b450dba0f73b927198e3e2292c2 not found: ID does not exist" Nov 03 22:19:49.221742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.221718 5931 scope.go:122] "RemoveContainer" containerID="5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2" Nov 03 22:19:49.230279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.230264 5931 scope.go:122] "RemoveContainer" containerID="5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2" Nov 03 22:19:49.230523 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:49.230503 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2\": container with ID starting with 5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2 not found: ID does not exist" containerID="5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2" Nov 03 22:19:49.230587 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.230529 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2"} err="failed to get container status \"5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2\": rpc error: code = NotFound desc = could not find container \"5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2\": container with ID starting with 5d3b28302cdb67ae7e6ccc660f6aa75ab618702440eaf304abdf465b078402c2 not found: ID does not exist" Nov 03 22:19:49.268168 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.268120 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5psk\" (UniqueName: \"kubernetes.io/projected/15288224-fbef-47db-b0a5-8d4f488a6915-kube-api-access-m5psk\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:49.356066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.356039 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z"] Nov 03 22:19:49.359926 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.359906 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-29dbd9-8c5674bff-cc26z"] Nov 03 22:19:49.466681 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.466650 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw"] Nov 03 22:19:49.469089 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:49.469078 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-29dbd9-84666d7684-gnkmw"] Nov 03 22:19:50.066717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:50.066635 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:50.505710 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:50.505682 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15288224-fbef-47db-b0a5-8d4f488a6915" path="/var/lib/kubelet/pods/15288224-fbef-47db-b0a5-8d4f488a6915/volumes" Nov 03 22:19:50.506067 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:50.506053 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85227cfc-1db8-46e0-964a-c24442389ad2" path="/var/lib/kubelet/pods/85227cfc-1db8-46e0-964a-c24442389ad2/volumes" Nov 03 22:19:53.497941 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.497905 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" podUID="d690c88c-dbed-4959-9c2c-722a6f41e8b6" containerName="payload-processing" containerID="cri-o://e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595" gracePeriod=25 Nov 03 22:19:53.607017 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.606988 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" podUID="b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" containerName="payload-pre-processing" containerID="cri-o://fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02" gracePeriod=25 Nov 03 22:19:53.764434 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.764365 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:19:53.807261 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.807232 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d690c88c-dbed-4959-9c2c-722a6f41e8b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d690c88c-dbed-4959-9c2c-722a6f41e8b6-plugins-config-volume\") pod \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " Nov 03 22:19:53.807406 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.807321 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d690c88c-dbed-4959-9c2c-722a6f41e8b6-kube-api-access-77xzs\" (UniqueName: \"kubernetes.io/projected/d690c88c-dbed-4959-9c2c-722a6f41e8b6-kube-api-access-77xzs\") pod \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\" (UID: \"d690c88c-dbed-4959-9c2c-722a6f41e8b6\") " Nov 03 22:19:53.807661 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.807635 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d690c88c-dbed-4959-9c2c-722a6f41e8b6-plugins-config-volume" pod "d690c88c-dbed-4959-9c2c-722a6f41e8b6" (UID: "d690c88c-dbed-4959-9c2c-722a6f41e8b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:19:53.807917 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.807898 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d690c88c-dbed-4959-9c2c-722a6f41e8b6-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:53.810114 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.810087 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d690c88c-dbed-4959-9c2c-722a6f41e8b6-kube-api-access-77xzs" pod "d690c88c-dbed-4959-9c2c-722a6f41e8b6" (UID: "d690c88c-dbed-4959-9c2c-722a6f41e8b6"). InnerVolumeSpecName "kube-api-access-77xzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:53.859783 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.859764 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:19:53.908372 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.908342 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-kube-api-access-f2vls\" (UniqueName: \"kubernetes.io/projected/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-kube-api-access-f2vls\") pod \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " Nov 03 22:19:53.908511 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.908460 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-plugins-config-volume\") pod \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\" (UID: \"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb\") " Nov 03 22:19:53.908801 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.908782 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77xzs\" (UniqueName: \"kubernetes.io/projected/d690c88c-dbed-4959-9c2c-722a6f41e8b6-kube-api-access-77xzs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:53.908915 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.908887 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-plugins-config-volume" pod "b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" (UID: "b74de4ef-bbf0-4916-bdb3-f2761fdf4abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:19:53.910514 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:53.910496 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-kube-api-access-f2vls" pod "b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" (UID: "b74de4ef-bbf0-4916-bdb3-f2761fdf4abb"). InnerVolumeSpecName "kube-api-access-f2vls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:54.009582 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.009555 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vls\" (UniqueName: \"kubernetes.io/projected/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-kube-api-access-f2vls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:54.009717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.009587 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:19:54.245030 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.245000 5931 generic.go:356] "Generic (PLEG): container finished" podID="d690c88c-dbed-4959-9c2c-722a6f41e8b6" containerID="e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595" exitCode=0 Nov 03 22:19:54.245199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.245063 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" Nov 03 22:19:54.245199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.245079 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" event={"ID":"d690c88c-dbed-4959-9c2c-722a6f41e8b6","Type":"ContainerDied","Data":"e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595"} Nov 03 22:19:54.245199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.245169 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk" event={"ID":"d690c88c-dbed-4959-9c2c-722a6f41e8b6","Type":"ContainerDied","Data":"941208d6dbb8d2885577273bc678bd8aa198ddb0811579f7b068950158e32e69"} Nov 03 22:19:54.245199 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.245194 5931 scope.go:122] "RemoveContainer" containerID="e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595" Nov 03 22:19:54.246854 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.246830 5931 generic.go:356] "Generic (PLEG): container finished" podID="b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" containerID="fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02" exitCode=0 Nov 03 22:19:54.246976 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.246857 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" Nov 03 22:19:54.246976 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.246902 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" event={"ID":"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb","Type":"ContainerDied","Data":"fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02"} Nov 03 22:19:54.246976 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.246956 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9" event={"ID":"b74de4ef-bbf0-4916-bdb3-f2761fdf4abb","Type":"ContainerDied","Data":"1cf31910cacabe9b2f974ba5aca6a18e38604502658ca7f56d79bd7e03a43bc2"} Nov 03 22:19:54.256881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.256804 5931 scope.go:122] "RemoveContainer" containerID="e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595" Nov 03 22:19:54.257194 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:54.257169 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595\": container with ID starting with e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595 not found: ID does not exist" containerID="e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595" Nov 03 22:19:54.257285 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.257203 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595"} err="failed to get container status \"e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595\": rpc error: code = NotFound desc = could not find container \"e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595\": container with ID starting with e4c37c2e95a6642fff56d4e020ad8fa0e711a29eeffb43a44426272eccc6a595 not found: ID does not exist" Nov 03 22:19:54.257285 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.257222 5931 scope.go:122] "RemoveContainer" containerID="fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02" Nov 03 22:19:54.265958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.265937 5931 scope.go:122] "RemoveContainer" containerID="fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02" Nov 03 22:19:54.266213 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:19:54.266195 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02\": container with ID starting with fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02 not found: ID does not exist" containerID="fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02" Nov 03 22:19:54.266275 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.266222 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02"} err="failed to get container status \"fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02\": rpc error: code = NotFound desc = could not find container \"fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02\": container with ID starting with fa19733305c9a9be501b89bc69e81cdd6fab6709dd996ac8b2c4ba9397c95b02 not found: ID does not exist" Nov 03 22:19:54.408397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.408372 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk"] Nov 03 22:19:54.410969 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.410952 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-534c77-5cd9b79b78-bz5qk"] Nov 03 22:19:54.505964 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.505913 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d690c88c-dbed-4959-9c2c-722a6f41e8b6" path="/var/lib/kubelet/pods/d690c88c-dbed-4959-9c2c-722a6f41e8b6/volumes" Nov 03 22:19:54.518184 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.518035 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9"] Nov 03 22:19:54.520675 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.520658 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-534c77-687dbb5479-l9ll9"] Nov 03 22:19:54.907420 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:54.907398 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:56.509126 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:56.509097 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" path="/var/lib/kubelet/pods/b74de4ef-bbf0-4916-bdb3-f2761fdf4abb/volumes" Nov 03 22:19:56.613428 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:56.613403 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:57.988699 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:57.988662 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:59.745668 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:19:59.745634 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:00.887244 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:00.887215 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:03.068349 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:20:03.068310 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:20:04.434447 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:04.434408 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:07.741674 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:07.741641 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:09.446958 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:09.446910 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:13.567877 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:13.567798 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:15.070851 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:15.070814 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:15.312501 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:15.312458 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:15.548342 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:15.548308 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:15.997875 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:15.997838 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:20.268694 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:20.268635 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:21.422187 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:21.422154 5931 generic.go:356] "Generic (PLEG): container finished" podID="30c1ac5d-4215-4817-af62-4931f675c3db" containerID="aecaacb4c87922822952a46b4c4791b451c9a2c7009728734c76dda9cd4a7d8e" exitCode=0 Nov 03 22:20:21.422635 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:21.422207 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" event={"ID":"30c1ac5d-4215-4817-af62-4931f675c3db","Type":"ContainerDied","Data":"aecaacb4c87922822952a46b4c4791b451c9a2c7009728734c76dda9cd4a7d8e"} Nov 03 22:20:21.422635 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:21.422495 5931 scope.go:122] "RemoveContainer" containerID="aecaacb4c87922822952a46b4c4791b451c9a2c7009728734c76dda9cd4a7d8e" Nov 03 22:20:21.423837 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:21.423809 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:20:21.423918 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:21.423873 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:20:21.423918 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:21.423903 5931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 22:20:21.933634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:21.932476 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:22.217084 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:22.217000 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:22.449596 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:22.449552 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:22.454263 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:22.454237 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" event={"ID":"30c1ac5d-4215-4817-af62-4931f675c3db","Type":"ContainerStarted","Data":"9d0832ab47317f2eaad4f51b34c8f3420e6a6258c61ba294777ba1ce64f34e0f"} Nov 03 22:20:23.343637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:23.338888 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:20:23.343637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:23.338992 5931 patch_prober.go:29] interesting pod/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:20:23.343637 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:23.339031 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 22:20:24.345011 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:24.344972 5931 patch_prober.go:29] interesting pod/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:20:24.346290 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:24.346256 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 22:20:25.001145 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:25.001103 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:25.340689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:25.328930 5931 patch_prober.go:29] interesting pod/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:20:25.340689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:25.328983 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 22:20:26.346634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:26.346084 5931 patch_prober.go:29] interesting pod/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:20:26.346634 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:26.346139 5931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 22:20:27.338540 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:27.338506 5931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:20:27.339061 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:27.339033 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:20:27.339702 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:27.339681 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:20:29.891139 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:29.891104 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:31.003261 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:31.003229 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:33.067683 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:20:33.067645 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:20:34.513538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:34.513505 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:35.879725 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:35.879696 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:20:35.884457 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:35.884434 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:20:36.348630 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:36.348589 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:36.577303 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:36.577273 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:39.249138 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:39.249098 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:43.394411 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:43.394372 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:44.906262 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:44.906215 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:45.137585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:45.137545 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:45.363704 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:45.363674 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:45.802702 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:45.802573 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:50.062076 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:50.062041 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:51.735280 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:51.735227 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:51.958234 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:51.958206 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:52.187769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:52.187735 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:55.259224 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:55.259192 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:58.359055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:58.359022 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:59.523835 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:20:59.523804 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:00.739525 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:00.739495 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:01.146296 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:01.146261 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:01.397047 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:01.396960 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:02.033860 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:02.033832 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:02.416847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:02.416816 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:02.648466 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:02.648430 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:02.893872 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:02.893836 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:03.067554 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:03.067521 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:21:03.414045 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:03.414019 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:05.537717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:05.537683 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj"] Nov 03 22:21:05.646231 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:05.646201 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47"] Nov 03 22:21:06.004802 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:06.004763 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:06.233851 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:06.233821 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:07.700446 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:07.700412 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:07.932217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:07.932186 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:08.164789 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:08.164756 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:09.215405 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:09.215375 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:09.444083 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:09.444048 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:09.675712 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:09.675680 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:10.102503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.102470 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:10.538563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.538501 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" podUID="59960395-b38c-4814-8fa6-8918fb2193f6" containerName="payload-processing" containerID="cri-o://930c314af8a1821fce13e950cbcb9cd4a25e081a65b7705817e2165e7aee0951" gracePeriod=25 Nov 03 22:21:10.647051 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.646951 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" podUID="72f4508e-ba50-469e-b9c6-60c528f613bc" containerName="payload-pre-processing" containerID="cri-o://d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df" gracePeriod=25 Nov 03 22:21:10.761354 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.761323 5931 generic.go:356] "Generic (PLEG): container finished" podID="59960395-b38c-4814-8fa6-8918fb2193f6" containerID="930c314af8a1821fce13e950cbcb9cd4a25e081a65b7705817e2165e7aee0951" exitCode=0 Nov 03 22:21:10.761603 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.761419 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" event={"ID":"59960395-b38c-4814-8fa6-8918fb2193f6","Type":"ContainerDied","Data":"930c314af8a1821fce13e950cbcb9cd4a25e081a65b7705817e2165e7aee0951"} Nov 03 22:21:10.817690 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.817668 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 22:21:10.931745 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.931428 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 22:21:10.945292 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.945264 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59960395-b38c-4814-8fa6-8918fb2193f6-kube-api-access-n9dsn\" (UniqueName: \"kubernetes.io/projected/59960395-b38c-4814-8fa6-8918fb2193f6-kube-api-access-n9dsn\") pod \"59960395-b38c-4814-8fa6-8918fb2193f6\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " Nov 03 22:21:10.945408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.945324 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59960395-b38c-4814-8fa6-8918fb2193f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59960395-b38c-4814-8fa6-8918fb2193f6-plugins-config-volume\") pod \"59960395-b38c-4814-8fa6-8918fb2193f6\" (UID: \"59960395-b38c-4814-8fa6-8918fb2193f6\") " Nov 03 22:21:10.945779 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.945742 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59960395-b38c-4814-8fa6-8918fb2193f6-plugins-config-volume" pod "59960395-b38c-4814-8fa6-8918fb2193f6" (UID: "59960395-b38c-4814-8fa6-8918fb2193f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:10.945871 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.945856 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59960395-b38c-4814-8fa6-8918fb2193f6-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:10.947543 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:10.947516 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59960395-b38c-4814-8fa6-8918fb2193f6-kube-api-access-n9dsn" pod "59960395-b38c-4814-8fa6-8918fb2193f6" (UID: "59960395-b38c-4814-8fa6-8918fb2193f6"). InnerVolumeSpecName "kube-api-access-n9dsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:11.046717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.046649 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f4508e-ba50-469e-b9c6-60c528f613bc-kube-api-access-bs7ll\" (UniqueName: \"kubernetes.io/projected/72f4508e-ba50-469e-b9c6-60c528f613bc-kube-api-access-bs7ll\") pod \"72f4508e-ba50-469e-b9c6-60c528f613bc\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " Nov 03 22:21:11.046855 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.046828 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f4508e-ba50-469e-b9c6-60c528f613bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f4508e-ba50-469e-b9c6-60c528f613bc-plugins-config-volume\") pod \"72f4508e-ba50-469e-b9c6-60c528f613bc\" (UID: \"72f4508e-ba50-469e-b9c6-60c528f613bc\") " Nov 03 22:21:11.047244 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.047223 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9dsn\" (UniqueName: \"kubernetes.io/projected/59960395-b38c-4814-8fa6-8918fb2193f6-kube-api-access-n9dsn\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:11.047584 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.047561 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f4508e-ba50-469e-b9c6-60c528f613bc-plugins-config-volume" pod "72f4508e-ba50-469e-b9c6-60c528f613bc" (UID: "72f4508e-ba50-469e-b9c6-60c528f613bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:11.049930 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.049909 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f4508e-ba50-469e-b9c6-60c528f613bc-kube-api-access-bs7ll" pod "72f4508e-ba50-469e-b9c6-60c528f613bc" (UID: "72f4508e-ba50-469e-b9c6-60c528f613bc"). InnerVolumeSpecName "kube-api-access-bs7ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:11.148087 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.148056 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7ll\" (UniqueName: \"kubernetes.io/projected/72f4508e-ba50-469e-b9c6-60c528f613bc-kube-api-access-bs7ll\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:11.148087 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.148081 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f4508e-ba50-469e-b9c6-60c528f613bc-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:11.768355 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.768327 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" Nov 03 22:21:11.768768 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.768354 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj" event={"ID":"59960395-b38c-4814-8fa6-8918fb2193f6","Type":"ContainerDied","Data":"b1150588eed89de39f03b011858a9d88af3ad1fcd07f14008ff47ba637e38c6f"} Nov 03 22:21:11.768768 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.768400 5931 scope.go:122] "RemoveContainer" containerID="930c314af8a1821fce13e950cbcb9cd4a25e081a65b7705817e2165e7aee0951" Nov 03 22:21:11.771874 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.771851 5931 generic.go:356] "Generic (PLEG): container finished" podID="72f4508e-ba50-469e-b9c6-60c528f613bc" containerID="d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df" exitCode=0 Nov 03 22:21:11.771976 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.771897 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" Nov 03 22:21:11.772047 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.771940 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" event={"ID":"72f4508e-ba50-469e-b9c6-60c528f613bc","Type":"ContainerDied","Data":"d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df"} Nov 03 22:21:11.772047 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.772016 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47" event={"ID":"72f4508e-ba50-469e-b9c6-60c528f613bc","Type":"ContainerDied","Data":"e4adea271269c1aa00e0812de451a452d2074a504b2286b591b0251c930a4ff2"} Nov 03 22:21:11.778782 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.778758 5931 scope.go:122] "RemoveContainer" containerID="d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df" Nov 03 22:21:11.787849 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.787834 5931 scope.go:122] "RemoveContainer" containerID="d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df" Nov 03 22:21:11.788096 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:11.788076 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df\": container with ID starting with d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df not found: ID does not exist" containerID="d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df" Nov 03 22:21:11.788165 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.788103 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df"} err="failed to get container status \"d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df\": rpc error: code = NotFound desc = could not find container \"d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df\": container with ID starting with d027bba33c5a040b3314f05f2bbd932b265086dc9c242f4696cb53b5238cd4df not found: ID does not exist" Nov 03 22:21:11.932148 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.932121 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47"] Nov 03 22:21:11.981717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:11.981695 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-87e1cf-85679cd5dc-j6t47"] Nov 03 22:21:12.044251 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:12.044228 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj"] Nov 03 22:21:12.056887 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:12.056865 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-87e1cf-89fccffd8-jkzcj"] Nov 03 22:21:12.243381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:12.243310 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:12.510638 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:12.510522 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59960395-b38c-4814-8fa6-8918fb2193f6" path="/var/lib/kubelet/pods/59960395-b38c-4814-8fa6-8918fb2193f6/volumes" Nov 03 22:21:12.511081 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:12.511065 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f4508e-ba50-469e-b9c6-60c528f613bc" path="/var/lib/kubelet/pods/72f4508e-ba50-469e-b9c6-60c528f613bc/volumes" Nov 03 22:21:13.880432 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:13.880395 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:14.110662 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:14.110630 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:14.334656 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:14.334547 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:14.827461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:14.827431 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:14.975595 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:14.975565 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj"] Nov 03 22:21:15.161013 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:15.160982 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:16.281655 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:16.281624 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:16.528297 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:16.528268 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:19.421087 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:19.421060 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:19.539242 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:19.539203 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd"] Nov 03 22:21:19.645780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:19.645750 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4"] Nov 03 22:21:20.031840 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:20.031807 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:20.276215 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:20.276183 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:20.786504 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:20.786471 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:20.982473 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:20.982438 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x"] Nov 03 22:21:20.982756 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:20.982726 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" containerID="cri-o://d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322" gracePeriod=30 Nov 03 22:21:21.027566 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:21.027536 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:21.268181 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:21.268151 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:23.754869 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:23.754838 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:23.982621 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:23.982590 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:24.539746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.539712 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" podUID="d2ca03b3-6b7c-4160-a9a1-9ebf98009784" containerName="payload-processing" containerID="cri-o://a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967" gracePeriod=25 Nov 03 22:21:24.589412 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.589384 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:24.646693 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.646636 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" podUID="09f06984-5f02-4e02-97fb-6d63c69fca38" containerName="payload-pre-processing" containerID="cri-o://734b816f1128833468d6505991a4a0e6ee7914a3883bc425e892c2cd8d9c0205" gracePeriod=25 Nov 03 22:21:24.810346 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.810319 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:21:24.859646 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.859588 5931 generic.go:356] "Generic (PLEG): container finished" podID="09f06984-5f02-4e02-97fb-6d63c69fca38" containerID="734b816f1128833468d6505991a4a0e6ee7914a3883bc425e892c2cd8d9c0205" exitCode=0 Nov 03 22:21:24.859777 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.859666 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" event={"ID":"09f06984-5f02-4e02-97fb-6d63c69fca38","Type":"ContainerDied","Data":"734b816f1128833468d6505991a4a0e6ee7914a3883bc425e892c2cd8d9c0205"} Nov 03 22:21:24.861270 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.861244 5931 generic.go:356] "Generic (PLEG): container finished" podID="d2ca03b3-6b7c-4160-a9a1-9ebf98009784" containerID="a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967" exitCode=0 Nov 03 22:21:24.861413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.861268 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" event={"ID":"d2ca03b3-6b7c-4160-a9a1-9ebf98009784","Type":"ContainerDied","Data":"a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967"} Nov 03 22:21:24.861413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.861308 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" Nov 03 22:21:24.861413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.861315 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd" event={"ID":"d2ca03b3-6b7c-4160-a9a1-9ebf98009784","Type":"ContainerDied","Data":"9407b531e36f956fe9a09396fbdc4aa55ab4c163430a9ea1d67d8a4ff551cf91"} Nov 03 22:21:24.861413 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.861343 5931 scope.go:122] "RemoveContainer" containerID="a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967" Nov 03 22:21:24.877217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.877196 5931 scope.go:122] "RemoveContainer" containerID="a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967" Nov 03 22:21:24.877546 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:24.877517 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967\": container with ID starting with a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967 not found: ID does not exist" containerID="a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967" Nov 03 22:21:24.877647 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.877569 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967"} err="failed to get container status \"a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967\": rpc error: code = NotFound desc = could not find container \"a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967\": container with ID starting with a18b3969cd3c5663e8f4273b47860ec606e1d1796bbb1f0aa666f160ccceb967 not found: ID does not exist" Nov 03 22:21:24.911304 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.911283 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:21:24.996974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.996935 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-kube-api-access-8kvwr\") pod \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " Nov 03 22:21:24.997180 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.997034 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f06984-5f02-4e02-97fb-6d63c69fca38-kube-api-access-zgtht\" (UniqueName: \"kubernetes.io/projected/09f06984-5f02-4e02-97fb-6d63c69fca38-kube-api-access-zgtht\") pod \"09f06984-5f02-4e02-97fb-6d63c69fca38\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " Nov 03 22:21:24.997180 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.997070 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-plugins-config-volume\") pod \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\" (UID: \"d2ca03b3-6b7c-4160-a9a1-9ebf98009784\") " Nov 03 22:21:24.997326 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.997169 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f06984-5f02-4e02-97fb-6d63c69fca38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f06984-5f02-4e02-97fb-6d63c69fca38-plugins-config-volume\") pod \"09f06984-5f02-4e02-97fb-6d63c69fca38\" (UID: \"09f06984-5f02-4e02-97fb-6d63c69fca38\") " Nov 03 22:21:24.997737 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.997687 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f06984-5f02-4e02-97fb-6d63c69fca38-plugins-config-volume" pod "09f06984-5f02-4e02-97fb-6d63c69fca38" (UID: "09f06984-5f02-4e02-97fb-6d63c69fca38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:24.997919 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.997895 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f06984-5f02-4e02-97fb-6d63c69fca38-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:24.998232 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.998206 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-plugins-config-volume" pod "d2ca03b3-6b7c-4160-a9a1-9ebf98009784" (UID: "d2ca03b3-6b7c-4160-a9a1-9ebf98009784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:24.999504 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.999483 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f06984-5f02-4e02-97fb-6d63c69fca38-kube-api-access-zgtht" pod "09f06984-5f02-4e02-97fb-6d63c69fca38" (UID: "09f06984-5f02-4e02-97fb-6d63c69fca38"). InnerVolumeSpecName "kube-api-access-zgtht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:24.999991 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:24.999534 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-kube-api-access-8kvwr" pod "d2ca03b3-6b7c-4160-a9a1-9ebf98009784" (UID: "d2ca03b3-6b7c-4160-a9a1-9ebf98009784"). InnerVolumeSpecName "kube-api-access-8kvwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:25.099012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.098981 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-kube-api-access-8kvwr\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:25.099209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.099016 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgtht\" (UniqueName: \"kubernetes.io/projected/09f06984-5f02-4e02-97fb-6d63c69fca38-kube-api-access-zgtht\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:25.099209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.099035 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ca03b3-6b7c-4160-a9a1-9ebf98009784-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:25.280410 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.280381 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd"] Nov 03 22:21:25.286304 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.286277 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0d45ba-6bfdbc6bbb-85dmd"] Nov 03 22:21:25.880660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.880626 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" Nov 03 22:21:25.880660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.880633 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4" event={"ID":"09f06984-5f02-4e02-97fb-6d63c69fca38","Type":"ContainerDied","Data":"819e9bb2e5fa5bc17b4df94c84299cfebf494ad3245765ea22674239ba055e6b"} Nov 03 22:21:25.881192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:25.880731 5931 scope.go:122] "RemoveContainer" containerID="734b816f1128833468d6505991a4a0e6ee7914a3883bc425e892c2cd8d9c0205" Nov 03 22:21:26.043360 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.043333 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4"] Nov 03 22:21:26.092956 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.092931 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0d45ba-8644cb855-66bk4"] Nov 03 22:21:26.243089 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.243069 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:21:26.312721 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.312696 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-socket\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.312864 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.312760 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-data\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.312864 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.312793 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-envoy\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.312984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.312903 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-podinfo\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.312984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.312936 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-token\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-token\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.312984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.312940 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-socket" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:26.312984 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.312965 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-certs\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.313187 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313018 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-data" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:26.313187 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313107 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06145d74-f6d7-4895-8e8e-ad0582b1f784-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06145d74-f6d7-4895-8e8e-ad0582b1f784-istiod-ca-cert\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.313187 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313163 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-kube-api-access-bkqdp\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-kube-api-access-bkqdp\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.313337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313193 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-credential-socket\") pod \"06145d74-f6d7-4895-8e8e-ad0582b1f784\" (UID: \"06145d74-f6d7-4895-8e8e-ad0582b1f784\") " Nov 03 22:21:26.313337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313250 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-certs" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:26.313520 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313495 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-credential-socket" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:26.313580 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313521 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06145d74-f6d7-4895-8e8e-ad0582b1f784-istiod-ca-cert" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:26.313824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313805 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.313876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313830 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06145d74-f6d7-4895-8e8e-ad0582b1f784-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.313876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313846 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.313876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313859 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.313876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.313873 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.315093 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.315066 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-podinfo" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:21:26.315186 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.315124 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-envoy" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:26.315186 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.315156 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-token" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:26.315795 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.315778 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-kube-api-access-bkqdp" pod "06145d74-f6d7-4895-8e8e-ad0582b1f784" (UID: "06145d74-f6d7-4895-8e8e-ad0582b1f784"). InnerVolumeSpecName "kube-api-access-bkqdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:26.414545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.414488 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqdp\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-kube-api-access-bkqdp\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.414545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.414513 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.414545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.414523 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.414545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.414531 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06145d74-f6d7-4895-8e8e-ad0582b1f784-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:26.506481 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.506452 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f06984-5f02-4e02-97fb-6d63c69fca38" path="/var/lib/kubelet/pods/09f06984-5f02-4e02-97fb-6d63c69fca38/volumes" Nov 03 22:21:26.506879 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.506864 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ca03b3-6b7c-4160-a9a1-9ebf98009784" path="/var/lib/kubelet/pods/d2ca03b3-6b7c-4160-a9a1-9ebf98009784/volumes" Nov 03 22:21:26.890639 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.890588 5931 generic.go:356] "Generic (PLEG): container finished" podID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerID="d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322" exitCode=0 Nov 03 22:21:26.891039 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.890662 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" Nov 03 22:21:26.891039 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.890676 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" event={"ID":"06145d74-f6d7-4895-8e8e-ad0582b1f784","Type":"ContainerDied","Data":"d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322"} Nov 03 22:21:26.891039 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.890733 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x" event={"ID":"06145d74-f6d7-4895-8e8e-ad0582b1f784","Type":"ContainerDied","Data":"06c548208a117748a043c1a60e314d394e6af948626db446fbd8fd84f4450872"} Nov 03 22:21:26.891039 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.890756 5931 scope.go:122] "RemoveContainer" containerID="d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322" Nov 03 22:21:26.900476 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.900458 5931 scope.go:122] "RemoveContainer" containerID="29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139" Nov 03 22:21:26.909239 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.909210 5931 scope.go:122] "RemoveContainer" containerID="d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322" Nov 03 22:21:26.909524 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:26.909507 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322\": container with ID starting with d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322 not found: ID does not exist" containerID="d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322" Nov 03 22:21:26.909585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.909540 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322"} err="failed to get container status \"d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322\": rpc error: code = NotFound desc = could not find container \"d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322\": container with ID starting with d51351082d1abddcc18b9b4853bfae7bea10fdea4074d44c16c921e77f2eb322 not found: ID does not exist" Nov 03 22:21:26.909585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.909557 5931 scope.go:122] "RemoveContainer" containerID="29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139" Nov 03 22:21:26.909815 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:26.909797 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139\": container with ID starting with 29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139 not found: ID does not exist" containerID="29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139" Nov 03 22:21:26.909856 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:26.909821 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139"} err="failed to get container status \"29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139\": rpc error: code = NotFound desc = could not find container \"29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139\": container with ID starting with 29d4512b9d92a832095bf0665eda4b51fa332cd6017e54726dd945edccbc3139 not found: ID does not exist" Nov 03 22:21:27.054785 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:27.054756 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x"] Nov 03 22:21:27.057550 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:27.057526 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a49a32-openshift-default-845d4d95b7-zzg2x"] Nov 03 22:21:28.506442 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:28.506410 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" path="/var/lib/kubelet/pods/06145d74-f6d7-4895-8e8e-ad0582b1f784/volumes" Nov 03 22:21:29.615139 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:29.615112 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:29.996897 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:29.996815 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" containerID="cri-o://a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f" gracePeriod=45 Nov 03 22:21:30.144213 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.144172 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 03 22:21:30.243518 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.243495 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 22:21:30.353561 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353541 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-home\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-home\") pod \"81838a1c-e57a-49e0-9286-5340e392a979\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " Nov 03 22:21:30.353693 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353588 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-model-cache\") pod \"81838a1c-e57a-49e0-9286-5340e392a979\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " Nov 03 22:21:30.353693 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353652 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-tmp-dir\") pod \"81838a1c-e57a-49e0-9286-5340e392a979\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " Nov 03 22:21:30.353693 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353678 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81838a1c-e57a-49e0-9286-5340e392a979-kube-api-access-jc77z\" (UniqueName: \"kubernetes.io/projected/81838a1c-e57a-49e0-9286-5340e392a979-kube-api-access-jc77z\") pod \"81838a1c-e57a-49e0-9286-5340e392a979\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " Nov 03 22:21:30.353813 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353715 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81838a1c-e57a-49e0-9286-5340e392a979-tls-certs\" (UniqueName: \"kubernetes.io/secret/81838a1c-e57a-49e0-9286-5340e392a979-tls-certs\") pod \"81838a1c-e57a-49e0-9286-5340e392a979\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " Nov 03 22:21:30.353865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353797 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-home" pod "81838a1c-e57a-49e0-9286-5340e392a979" (UID: "81838a1c-e57a-49e0-9286-5340e392a979"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:30.353865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353822 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-dshm\") pod \"81838a1c-e57a-49e0-9286-5340e392a979\" (UID: \"81838a1c-e57a-49e0-9286-5340e392a979\") " Nov 03 22:21:30.353970 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353935 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-model-cache" pod "81838a1c-e57a-49e0-9286-5340e392a979" (UID: "81838a1c-e57a-49e0-9286-5340e392a979"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:30.353970 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.353951 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-tmp-dir" pod "81838a1c-e57a-49e0-9286-5340e392a979" (UID: "81838a1c-e57a-49e0-9286-5340e392a979"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:30.354192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.354173 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:30.354192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.354193 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:30.354347 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.354206 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:30.356648 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.356624 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81838a1c-e57a-49e0-9286-5340e392a979-tls-certs" pod "81838a1c-e57a-49e0-9286-5340e392a979" (UID: "81838a1c-e57a-49e0-9286-5340e392a979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:21:30.356791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.356669 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81838a1c-e57a-49e0-9286-5340e392a979-kube-api-access-jc77z" pod "81838a1c-e57a-49e0-9286-5340e392a979" (UID: "81838a1c-e57a-49e0-9286-5340e392a979"). InnerVolumeSpecName "kube-api-access-jc77z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:30.356791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.356691 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-dshm" pod "81838a1c-e57a-49e0-9286-5340e392a979" (UID: "81838a1c-e57a-49e0-9286-5340e392a979"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:21:30.455396 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.455366 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc77z\" (UniqueName: \"kubernetes.io/projected/81838a1c-e57a-49e0-9286-5340e392a979-kube-api-access-jc77z\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:30.455396 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.455394 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81838a1c-e57a-49e0-9286-5340e392a979-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:30.455578 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.455404 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81838a1c-e57a-49e0-9286-5340e392a979-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:30.917219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.917187 5931 generic.go:356] "Generic (PLEG): container finished" podID="81838a1c-e57a-49e0-9286-5340e392a979" containerID="a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f" exitCode=0 Nov 03 22:21:30.917598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.917227 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" event={"ID":"81838a1c-e57a-49e0-9286-5340e392a979","Type":"ContainerDied","Data":"a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f"} Nov 03 22:21:30.917598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.917247 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" Nov 03 22:21:30.917598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.917271 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj" event={"ID":"81838a1c-e57a-49e0-9286-5340e392a979","Type":"ContainerDied","Data":"8b4ffdb8584f0d2b02a597f650a3e13e27b331595fcebd2c3dba01a1d5449c73"} Nov 03 22:21:30.917598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.917289 5931 scope.go:122] "RemoveContainer" containerID="a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f" Nov 03 22:21:30.927021 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.927006 5931 scope.go:122] "RemoveContainer" containerID="a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f" Nov 03 22:21:30.927283 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:30.927263 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f\": container with ID starting with a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f not found: ID does not exist" containerID="a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f" Nov 03 22:21:30.927354 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:30.927292 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f"} err="failed to get container status \"a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f\": rpc error: code = NotFound desc = could not find container \"a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f\": container with ID starting with a0ebf1e72af79b487fb58c480f305b697c7e1bc36bcc40bdcf741fd5eb2c134f not found: ID does not exist" Nov 03 22:21:31.079444 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:31.079422 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj"] Nov 03 22:21:31.128444 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:31.128372 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-w1-f463c9-kserve-86f5c7f697-x6ssj"] Nov 03 22:21:32.505405 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:32.505371 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81838a1c-e57a-49e0-9286-5340e392a979" path="/var/lib/kubelet/pods/81838a1c-e57a-49e0-9286-5340e392a979/volumes" Nov 03 22:21:33.067959 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:33.067932 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:21:33.472115 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:33.472080 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7"] Nov 03 22:21:33.579013 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:33.578985 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t"] Nov 03 22:21:33.944824 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:33.944795 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:34.175520 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:34.175489 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:34.631742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:34.631715 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:38.473428 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.473380 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" podUID="18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" containerName="payload-processing" containerID="cri-o://d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d" gracePeriod=25 Nov 03 22:21:38.579742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.579710 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" podUID="0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" containerName="payload-pre-processing" containerID="cri-o://4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f" gracePeriod=25 Nov 03 22:21:38.719929 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.719904 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:21:38.830685 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.830660 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-plugins-config-volume\") pod \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " Nov 03 22:21:38.830790 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.830700 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-kube-api-access-xl8pt\" (UniqueName: \"kubernetes.io/projected/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-kube-api-access-xl8pt\") pod \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\" (UID: \"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a\") " Nov 03 22:21:38.831045 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.831023 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-plugins-config-volume" pod "18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" (UID: "18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:38.831271 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.831257 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:38.833160 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.833120 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-kube-api-access-xl8pt" pod "18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" (UID: "18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a"). InnerVolumeSpecName "kube-api-access-xl8pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:38.838543 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.838520 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:21:38.931998 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.931970 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-plugins-config-volume\") pod \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " Nov 03 22:21:38.932134 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.932066 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-kube-api-access-9wgdc\" (UniqueName: \"kubernetes.io/projected/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-kube-api-access-9wgdc\") pod \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\" (UID: \"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f\") " Nov 03 22:21:38.932358 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.932333 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-plugins-config-volume" pod "0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" (UID: "0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:38.932431 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.932403 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl8pt\" (UniqueName: \"kubernetes.io/projected/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a-kube-api-access-xl8pt\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:38.934083 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.934068 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-kube-api-access-9wgdc" pod "0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" (UID: "0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f"). InnerVolumeSpecName "kube-api-access-9wgdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:38.982542 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.982488 5931 generic.go:356] "Generic (PLEG): container finished" podID="18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" containerID="d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d" exitCode=0 Nov 03 22:21:38.982542 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.982537 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" Nov 03 22:21:38.982758 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.982570 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" event={"ID":"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a","Type":"ContainerDied","Data":"d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d"} Nov 03 22:21:38.982758 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.982670 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7" event={"ID":"18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a","Type":"ContainerDied","Data":"3467d314ff303a4f6f14c2436f33469078f7b85571eafa385d4d6dc3f8974a86"} Nov 03 22:21:38.982758 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.982714 5931 scope.go:122] "RemoveContainer" containerID="d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d" Nov 03 22:21:38.984357 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.984330 5931 generic.go:356] "Generic (PLEG): container finished" podID="0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" containerID="4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f" exitCode=0 Nov 03 22:21:38.984474 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.984388 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" Nov 03 22:21:38.984549 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.984515 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" event={"ID":"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f","Type":"ContainerDied","Data":"4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f"} Nov 03 22:21:38.984651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.984549 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t" event={"ID":"0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f","Type":"ContainerDied","Data":"3f9301e17f68f910aa3eecf91fb873e28f370cfab9c3cc4b0db2a16d2d3a6114"} Nov 03 22:21:38.995535 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.995519 5931 scope.go:122] "RemoveContainer" containerID="d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d" Nov 03 22:21:38.995816 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:38.995794 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d\": container with ID starting with d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d not found: ID does not exist" containerID="d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d" Nov 03 22:21:38.995914 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.995822 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d"} err="failed to get container status \"d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d\": rpc error: code = NotFound desc = could not find container \"d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d\": container with ID starting with d765778ef7cac4b2559d7589f2f880bcd60d032460ba53872c59a2d5a3826b9d not found: ID does not exist" Nov 03 22:21:38.995914 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:38.995839 5931 scope.go:122] "RemoveContainer" containerID="4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f" Nov 03 22:21:39.007465 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.007445 5931 scope.go:122] "RemoveContainer" containerID="4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f" Nov 03 22:21:39.007789 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:39.007763 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f\": container with ID starting with 4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f not found: ID does not exist" containerID="4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f" Nov 03 22:21:39.007881 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.007796 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f"} err="failed to get container status \"4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f\": rpc error: code = NotFound desc = could not find container \"4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f\": container with ID starting with 4af1c6b11fd18747465a8f556be39c86789760361fc0144752c8766ad8e8369f not found: ID does not exist" Nov 03 22:21:39.033695 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.033676 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wgdc\" (UniqueName: \"kubernetes.io/projected/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-kube-api-access-9wgdc\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:39.033771 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.033697 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:39.144917 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.144894 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7"] Nov 03 22:21:39.148071 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.148052 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75f577-6769bfdf7b-d4sv7"] Nov 03 22:21:39.255838 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.255809 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t"] Nov 03 22:21:39.259596 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.259539 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75f577-7bff446fc9-4fs9t"] Nov 03 22:21:39.272786 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:39.272764 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:40.505712 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:40.505684 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" path="/var/lib/kubelet/pods/0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f/volumes" Nov 03 22:21:40.506099 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:40.506032 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" path="/var/lib/kubelet/pods/18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a/volumes" Nov 03 22:21:43.361849 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:43.361774 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:44.856827 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:44.856794 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:45.136746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:45.136673 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:45.383899 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:45.383867 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:45.875742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:45.875710 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:50.554669 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:50.554603 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:52.232978 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:52.232907 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:52.488977 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:52.488900 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:52.731032 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:52.731004 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:52.957197 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:52.957169 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:54.718060 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:54.718023 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz"] Nov 03 22:21:54.718491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:54.718206 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" podUID="479afc9c-4f61-40bb-8b23-5de937c635c3" containerName="maas-api" containerID="cri-o://e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1" gracePeriod=30 Nov 03 22:21:54.911695 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:54.911665 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:54.970780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:54.970721 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:21:55.080902 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.080876 5931 generic.go:356] "Generic (PLEG): container finished" podID="479afc9c-4f61-40bb-8b23-5de937c635c3" containerID="e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1" exitCode=0 Nov 03 22:21:55.081066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.080916 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" Nov 03 22:21:55.081066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.080966 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" event={"ID":"479afc9c-4f61-40bb-8b23-5de937c635c3","Type":"ContainerDied","Data":"e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1"} Nov 03 22:21:55.081066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.081025 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz" event={"ID":"479afc9c-4f61-40bb-8b23-5de937c635c3","Type":"ContainerDied","Data":"c34c5f95fd8a705a6b478c505bc1c8451f6b1cdf26ee85442e4786191b8b7fe3"} Nov 03 22:21:55.081066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.081055 5931 scope.go:122] "RemoveContainer" containerID="e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1" Nov 03 22:21:55.083383 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.083359 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-metrics-certs\") pod \"479afc9c-4f61-40bb-8b23-5de937c635c3\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " Nov 03 22:21:55.083493 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.083394 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls\") pod \"479afc9c-4f61-40bb-8b23-5de937c635c3\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " Nov 03 22:21:55.083493 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.083457 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479afc9c-4f61-40bb-8b23-5de937c635c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/479afc9c-4f61-40bb-8b23-5de937c635c3-openshift-service-ca\") pod \"479afc9c-4f61-40bb-8b23-5de937c635c3\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " Nov 03 22:21:55.083646 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.083494 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479afc9c-4f61-40bb-8b23-5de937c635c3-kube-api-access-xx54s\" (UniqueName: \"kubernetes.io/projected/479afc9c-4f61-40bb-8b23-5de937c635c3-kube-api-access-xx54s\") pod \"479afc9c-4f61-40bb-8b23-5de937c635c3\" (UID: \"479afc9c-4f61-40bb-8b23-5de937c635c3\") " Nov 03 22:21:55.083851 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.083832 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479afc9c-4f61-40bb-8b23-5de937c635c3-openshift-service-ca" pod "479afc9c-4f61-40bb-8b23-5de937c635c3" (UID: "479afc9c-4f61-40bb-8b23-5de937c635c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:21:55.084121 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.084052 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/479afc9c-4f61-40bb-8b23-5de937c635c3-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:55.086426 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.086398 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479afc9c-4f61-40bb-8b23-5de937c635c3-kube-api-access-xx54s" pod "479afc9c-4f61-40bb-8b23-5de937c635c3" (UID: "479afc9c-4f61-40bb-8b23-5de937c635c3"). InnerVolumeSpecName "kube-api-access-xx54s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:55.086542 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.086483 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls" pod "479afc9c-4f61-40bb-8b23-5de937c635c3" (UID: "479afc9c-4f61-40bb-8b23-5de937c635c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:21:55.086849 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.086822 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-metrics-certs" pod "479afc9c-4f61-40bb-8b23-5de937c635c3" (UID: "479afc9c-4f61-40bb-8b23-5de937c635c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:21:55.118891 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.118841 5931 scope.go:122] "RemoveContainer" containerID="e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1" Nov 03 22:21:55.121186 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:21:55.121108 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1\": container with ID starting with e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1 not found: ID does not exist" containerID="e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1" Nov 03 22:21:55.121186 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.121150 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1"} err="failed to get container status \"e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1\": rpc error: code = NotFound desc = could not find container \"e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1\": container with ID starting with e38c66f20dde38f14da576ddcbfa8c08addfd4b4f30f86bb61c4873c810252a1 not found: ID does not exist" Nov 03 22:21:55.185094 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.185069 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx54s\" (UniqueName: \"kubernetes.io/projected/479afc9c-4f61-40bb-8b23-5de937c635c3-kube-api-access-xx54s\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:55.185094 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.185094 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:55.185232 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.185103 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/479afc9c-4f61-40bb-8b23-5de937c635c3-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:21:55.503819 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.503778 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz"] Nov 03 22:21:55.506012 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:55.505986 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-48ae0a-787965cd58-njcvz"] Nov 03 22:21:56.455136 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:56.455107 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:56.505743 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:56.505719 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479afc9c-4f61-40bb-8b23-5de937c635c3" path="/var/lib/kubelet/pods/479afc9c-4f61-40bb-8b23-5de937c635c3/volumes" Nov 03 22:21:56.771307 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:56.771232 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:56.997960 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:56.997933 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:57.378890 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:57.378836 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:58.646903 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:58.646876 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:58.884934 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:21:58.884909 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:01.266335 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:01.266298 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:02.827904 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:02.827874 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:03.063134 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:03.063105 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:03.073159 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:03.073123 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:22:03.314177 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:03.314147 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:03.686969 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:03.686927 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:03.920065 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:03.920037 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:03.926582 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:03.926554 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q"] Nov 03 22:22:03.926810 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:03.926788 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" containerID="cri-o://9d0832ab47317f2eaad4f51b34c8f3420e6a6258c61ba294777ba1ce64f34e0f" gracePeriod=30 Nov 03 22:22:04.157471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:04.157435 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:04.915528 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:04.915494 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9"] Nov 03 22:22:05.024229 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:05.024203 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr"] Nov 03 22:22:05.369778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:05.369734 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:05.593750 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:05.593720 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:08.276515 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:08.276484 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:09.169869 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.169842 5931 generic.go:356] "Generic (PLEG): container finished" podID="30c1ac5d-4215-4817-af62-4931f675c3db" containerID="9d0832ab47317f2eaad4f51b34c8f3420e6a6258c61ba294777ba1ce64f34e0f" exitCode=0 Nov 03 22:22:09.170000 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.169897 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" event={"ID":"30c1ac5d-4215-4817-af62-4931f675c3db","Type":"ContainerDied","Data":"9d0832ab47317f2eaad4f51b34c8f3420e6a6258c61ba294777ba1ce64f34e0f"} Nov 03 22:22:09.170000 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.169964 5931 scope.go:122] "RemoveContainer" containerID="aecaacb4c87922822952a46b4c4791b451c9a2c7009728734c76dda9cd4a7d8e" Nov 03 22:22:09.189968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.189947 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:22:09.227102 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227049 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-certs\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227102 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227087 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-data\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227208 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-kube-api-access-4ps4n\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-kube-api-access-4ps4n\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227331 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227292 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-socket\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227331 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227317 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-certs" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:09.227441 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227338 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c1ac5d-4215-4817-af62-4931f675c3db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1ac5d-4215-4817-af62-4931f675c3db-istiod-ca-cert\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227441 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227389 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-istio-token\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-istio-token\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227441 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227401 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-data" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:09.227441 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227434 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30c1ac5d-4215-4817-af62-4931f675c3db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1ac5d-4215-4817-af62-4931f675c3db-istio-podinfo\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227703 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227492 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-envoy\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227703 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227536 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-credential-socket\") pod \"30c1ac5d-4215-4817-af62-4931f675c3db\" (UID: \"30c1ac5d-4215-4817-af62-4931f675c3db\") " Nov 03 22:22:09.227703 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227555 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-socket" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:09.227874 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227722 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1ac5d-4215-4817-af62-4931f675c3db-istiod-ca-cert" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:09.227874 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.227813 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-credential-socket" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:09.228063 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.228043 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.228111 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.228071 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.228194 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.228117 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.228271 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.228259 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.228335 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.228275 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1ac5d-4215-4817-af62-4931f675c3db-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.229454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.229432 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-kube-api-access-4ps4n" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "kube-api-access-4ps4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:09.229747 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.229714 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30c1ac5d-4215-4817-af62-4931f675c3db-istio-podinfo" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:22:09.229892 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.229869 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-envoy" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:09.229980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.229962 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-istio-token" pod "30c1ac5d-4215-4817-af62-4931f675c3db" (UID: "30c1ac5d-4215-4817-af62-4931f675c3db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:09.329297 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.329271 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.329297 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.329294 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1ac5d-4215-4817-af62-4931f675c3db-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.329297 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.329304 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1ac5d-4215-4817-af62-4931f675c3db-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.329696 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.329313 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ps4n\" (UniqueName: \"kubernetes.io/projected/30c1ac5d-4215-4817-af62-4931f675c3db-kube-api-access-4ps4n\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:09.861821 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.861773 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:09.916598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:09.916563 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" podUID="fc52c97f-0de4-4af8-acb3-7da8534e527c" containerName="payload-processing" containerID="cri-o://ddde129ce44ad11bd1283ddc80c8d1c4896863d9fe66671f563e63873779e395" gracePeriod=25 Nov 03 22:22:10.025179 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.025112 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" podUID="3addd93b-3e66-469f-bd37-267517de0347" containerName="payload-pre-processing" containerID="cri-o://e62164a1db8d4a6139c366daf6016ce5dd845547f4a3989140b4cd1328678c52" gracePeriod=25 Nov 03 22:22:10.092776 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.092678 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:10.180809 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.180773 5931 generic.go:356] "Generic (PLEG): container finished" podID="fc52c97f-0de4-4af8-acb3-7da8534e527c" containerID="ddde129ce44ad11bd1283ddc80c8d1c4896863d9fe66671f563e63873779e395" exitCode=0 Nov 03 22:22:10.180941 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.180850 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" event={"ID":"fc52c97f-0de4-4af8-acb3-7da8534e527c","Type":"ContainerDied","Data":"ddde129ce44ad11bd1283ddc80c8d1c4896863d9fe66671f563e63873779e395"} Nov 03 22:22:10.185651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.182655 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" event={"ID":"30c1ac5d-4215-4817-af62-4931f675c3db","Type":"ContainerDied","Data":"5731e369c65e826132d223e955dc084a89d0a73dd549bf4d0ba03c9ff9a21a2f"} Nov 03 22:22:10.185651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.182696 5931 scope.go:122] "RemoveContainer" containerID="9d0832ab47317f2eaad4f51b34c8f3420e6a6258c61ba294777ba1ce64f34e0f" Nov 03 22:22:10.185651 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.182780 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q" Nov 03 22:22:10.187442 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.187417 5931 generic.go:356] "Generic (PLEG): container finished" podID="3addd93b-3e66-469f-bd37-267517de0347" containerID="e62164a1db8d4a6139c366daf6016ce5dd845547f4a3989140b4cd1328678c52" exitCode=0 Nov 03 22:22:10.187576 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.187451 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" event={"ID":"3addd93b-3e66-469f-bd37-267517de0347","Type":"ContainerDied","Data":"e62164a1db8d4a6139c366daf6016ce5dd845547f4a3989140b4cd1328678c52"} Nov 03 22:22:10.205759 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.205742 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 22:22:10.236989 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.236968 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc52c97f-0de4-4af8-acb3-7da8534e527c-kube-api-access-6hgp8\" (UniqueName: \"kubernetes.io/projected/fc52c97f-0de4-4af8-acb3-7da8534e527c-kube-api-access-6hgp8\") pod \"fc52c97f-0de4-4af8-acb3-7da8534e527c\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " Nov 03 22:22:10.237141 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.237130 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc52c97f-0de4-4af8-acb3-7da8534e527c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc52c97f-0de4-4af8-acb3-7da8534e527c-plugins-config-volume\") pod \"fc52c97f-0de4-4af8-acb3-7da8534e527c\" (UID: \"fc52c97f-0de4-4af8-acb3-7da8534e527c\") " Nov 03 22:22:10.237495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.237454 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc52c97f-0de4-4af8-acb3-7da8534e527c-plugins-config-volume" pod "fc52c97f-0de4-4af8-acb3-7da8534e527c" (UID: "fc52c97f-0de4-4af8-acb3-7da8534e527c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:10.241478 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.241456 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc52c97f-0de4-4af8-acb3-7da8534e527c-kube-api-access-6hgp8" pod "fc52c97f-0de4-4af8-acb3-7da8534e527c" (UID: "fc52c97f-0de4-4af8-acb3-7da8534e527c"). InnerVolumeSpecName "kube-api-access-6hgp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:10.298251 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.298229 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 22:22:10.321902 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.321877 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:10.338647 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.338605 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3addd93b-3e66-469f-bd37-267517de0347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3addd93b-3e66-469f-bd37-267517de0347-plugins-config-volume\") pod \"3addd93b-3e66-469f-bd37-267517de0347\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " Nov 03 22:22:10.338923 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.338740 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3addd93b-3e66-469f-bd37-267517de0347-kube-api-access-l55wp\" (UniqueName: \"kubernetes.io/projected/3addd93b-3e66-469f-bd37-267517de0347-kube-api-access-l55wp\") pod \"3addd93b-3e66-469f-bd37-267517de0347\" (UID: \"3addd93b-3e66-469f-bd37-267517de0347\") " Nov 03 22:22:10.338923 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.338872 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3addd93b-3e66-469f-bd37-267517de0347-plugins-config-volume" pod "3addd93b-3e66-469f-bd37-267517de0347" (UID: "3addd93b-3e66-469f-bd37-267517de0347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:10.339076 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.339053 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc52c97f-0de4-4af8-acb3-7da8534e527c-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:10.339076 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.339076 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgp8\" (UniqueName: \"kubernetes.io/projected/fc52c97f-0de4-4af8-acb3-7da8534e527c-kube-api-access-6hgp8\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:10.339222 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.339085 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3addd93b-3e66-469f-bd37-267517de0347-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:10.340699 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.340679 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3addd93b-3e66-469f-bd37-267517de0347-kube-api-access-l55wp" pod "3addd93b-3e66-469f-bd37-267517de0347" (UID: "3addd93b-3e66-469f-bd37-267517de0347"). InnerVolumeSpecName "kube-api-access-l55wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:10.439999 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.439951 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55wp\" (UniqueName: \"kubernetes.io/projected/3addd93b-3e66-469f-bd37-267517de0347-kube-api-access-l55wp\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:10.462945 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.462915 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q"] Nov 03 22:22:10.468134 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.468115 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-48ae0a-openshift-default-74c54874d5-ptm9q"] Nov 03 22:22:10.505805 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.505785 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" path="/var/lib/kubelet/pods/30c1ac5d-4215-4817-af62-4931f675c3db/volumes" Nov 03 22:22:10.648598 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:10.648565 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:11.194198 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.194169 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" Nov 03 22:22:11.194198 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.194173 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr" event={"ID":"3addd93b-3e66-469f-bd37-267517de0347","Type":"ContainerDied","Data":"c7ef8478920bd750057e149b2eb7f41a547dac4f779cfedc990e5098aa773b3c"} Nov 03 22:22:11.194435 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.194239 5931 scope.go:122] "RemoveContainer" containerID="e62164a1db8d4a6139c366daf6016ce5dd845547f4a3989140b4cd1328678c52" Nov 03 22:22:11.195732 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.195715 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" Nov 03 22:22:11.195843 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.195739 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9" event={"ID":"fc52c97f-0de4-4af8-acb3-7da8534e527c","Type":"ContainerDied","Data":"492d324d762aa4b31fd4a3720ee1f9fb98242a7b9829b2af2ed79fa1985eff35"} Nov 03 22:22:11.204088 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.204074 5931 scope.go:122] "RemoveContainer" containerID="ddde129ce44ad11bd1283ddc80c8d1c4896863d9fe66671f563e63873779e395" Nov 03 22:22:11.356536 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.356513 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9"] Nov 03 22:22:11.360397 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.360379 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6ddbba-6d97d85684-qq8g9"] Nov 03 22:22:11.469305 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.469283 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr"] Nov 03 22:22:11.471362 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:11.471343 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6ddbba-6bc5dbbb75-7wphr"] Nov 03 22:22:12.505354 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:12.505324 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3addd93b-3e66-469f-bd37-267517de0347" path="/var/lib/kubelet/pods/3addd93b-3e66-469f-bd37-267517de0347/volumes" Nov 03 22:22:12.505742 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:12.505682 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc52c97f-0de4-4af8-acb3-7da8534e527c" path="/var/lib/kubelet/pods/fc52c97f-0de4-4af8-acb3-7da8534e527c/volumes" Nov 03 22:22:12.923419 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:12.923374 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:14.459644 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:14.459591 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:14.691438 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:14.691410 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:14.917259 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:14.917234 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:15.400819 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:15.400785 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:15.768255 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:15.768185 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:18.113857 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:18.113804 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:18.339733 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:18.339699 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:19.795271 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:19.795237 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:21.138189 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:21.138156 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:21.363252 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:21.363225 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:21.372864 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:21.372840 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55"] Nov 03 22:22:21.373053 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:21.373033 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" containerID="cri-o://50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c" gracePeriod=30 Nov 03 22:22:21.589141 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:21.589070 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:21.946781 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:21.946753 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:24.225265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:24.225224 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:24.460547 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:24.460517 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:24.954802 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:24.954777 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:26.624752 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.624728 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:22:26.688887 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.688831 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-envoy\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.688991 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.688922 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-certs\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.688991 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.688946 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-credential-socket\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.688991 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.688964 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-data\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.689126 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689005 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-token\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-token\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.689126 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689083 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-kube-api-access-v8nbh\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-kube-api-access-v8nbh\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.689126 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689110 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-socket\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.689289 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689166 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-certs" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:26.689289 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689244 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-credential-socket" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:26.689289 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689260 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-data" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:26.689464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689333 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-socket" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:26.689464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689372 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-podinfo\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.689464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689441 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c27741f-5733-4eae-92a1-40eb8fc93976-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c27741f-5733-4eae-92a1-40eb8fc93976-istiod-ca-cert\") pod \"1c27741f-5733-4eae-92a1-40eb8fc93976\" (UID: \"1c27741f-5733-4eae-92a1-40eb8fc93976\") " Nov 03 22:22:26.689863 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.689798 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c27741f-5733-4eae-92a1-40eb8fc93976-istiod-ca-cert" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:26.690029 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.690005 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.690109 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.690035 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.690109 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.690050 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.690109 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.690064 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.690109 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.690077 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c27741f-5733-4eae-92a1-40eb8fc93976-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.691166 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.691142 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-envoy" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:26.691284 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.691260 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-kube-api-access-v8nbh" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "kube-api-access-v8nbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:26.691338 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.691294 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-token" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:26.691384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.691360 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-podinfo" pod "1c27741f-5733-4eae-92a1-40eb8fc93976" (UID: "1c27741f-5733-4eae-92a1-40eb8fc93976"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:22:26.792262 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.792236 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8nbh\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-kube-api-access-v8nbh\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.792386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.792267 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.792386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.792281 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.792386 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.792294 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c27741f-5733-4eae-92a1-40eb8fc93976-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:26.841563 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.841524 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq"] Nov 03 22:22:26.994544 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:26.994461 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:27.289521 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.289452 5931 generic.go:356] "Generic (PLEG): container finished" podID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerID="50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c" exitCode=0 Nov 03 22:22:27.289521 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.289486 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" event={"ID":"1c27741f-5733-4eae-92a1-40eb8fc93976","Type":"ContainerDied","Data":"50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c"} Nov 03 22:22:27.289791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.289517 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" Nov 03 22:22:27.289791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.289541 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55" event={"ID":"1c27741f-5733-4eae-92a1-40eb8fc93976","Type":"ContainerDied","Data":"24cf47b213977e76b2b1349d2762a0ad85904a65f3e67792672d81bac4166353"} Nov 03 22:22:27.289791 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.289566 5931 scope.go:122] "RemoveContainer" containerID="50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c" Nov 03 22:22:27.300449 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.300432 5931 scope.go:122] "RemoveContainer" containerID="3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39" Nov 03 22:22:27.309194 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.309178 5931 scope.go:122] "RemoveContainer" containerID="50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c" Nov 03 22:22:27.309479 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:27.309460 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c\": container with ID starting with 50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c not found: ID does not exist" containerID="50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c" Nov 03 22:22:27.309545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.309485 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c"} err="failed to get container status \"50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c\": rpc error: code = NotFound desc = could not find container \"50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c\": container with ID starting with 50420f519cefad62bd9b2d44623cd38a0a9c1c0655dd06cadcc6d8195bff9d9c not found: ID does not exist" Nov 03 22:22:27.309545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.309502 5931 scope.go:122] "RemoveContainer" containerID="3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39" Nov 03 22:22:27.309758 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:27.309743 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39\": container with ID starting with 3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39 not found: ID does not exist" containerID="3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39" Nov 03 22:22:27.309817 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.309763 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39"} err="failed to get container status \"3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39\": rpc error: code = NotFound desc = could not find container \"3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39\": container with ID starting with 3029e68e815e775dee13f7b69011f6342d161d0b99de47af086c2f319b3cbe39 not found: ID does not exist" Nov 03 22:22:27.452956 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.452930 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55"] Nov 03 22:22:27.501996 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:27.501974 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75f577-openshift-default-54d4f58f5f-djp55"] Nov 03 22:22:28.505337 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:28.505298 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" path="/var/lib/kubelet/pods/1c27741f-5733-4eae-92a1-40eb8fc93976/volumes" Nov 03 22:22:29.517172 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:29.517112 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:30.510565 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.510535 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t"] Nov 03 22:22:30.510763 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.510730 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" podUID="cc07e711-7215-44f0-86b5-69a682869a4d" containerName="maas-api" containerID="cri-o://6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b" gracePeriod=30 Nov 03 22:22:30.703020 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.702991 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:30.761177 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.761127 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 22:22:30.928072 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.928037 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc07e711-7215-44f0-86b5-69a682869a4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc07e711-7215-44f0-86b5-69a682869a4d-openshift-service-ca\") pod \"cc07e711-7215-44f0-86b5-69a682869a4d\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " Nov 03 22:22:30.928217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.928155 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-maas-api-tls\") pod \"cc07e711-7215-44f0-86b5-69a682869a4d\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " Nov 03 22:22:30.928217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.928184 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-metrics-certs\") pod \"cc07e711-7215-44f0-86b5-69a682869a4d\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " Nov 03 22:22:30.928301 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.928242 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc07e711-7215-44f0-86b5-69a682869a4d-kube-api-access-rrstw\" (UniqueName: \"kubernetes.io/projected/cc07e711-7215-44f0-86b5-69a682869a4d-kube-api-access-rrstw\") pod \"cc07e711-7215-44f0-86b5-69a682869a4d\" (UID: \"cc07e711-7215-44f0-86b5-69a682869a4d\") " Nov 03 22:22:30.928402 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.928379 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc07e711-7215-44f0-86b5-69a682869a4d-openshift-service-ca" pod "cc07e711-7215-44f0-86b5-69a682869a4d" (UID: "cc07e711-7215-44f0-86b5-69a682869a4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:30.928669 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.928649 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc07e711-7215-44f0-86b5-69a682869a4d-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:30.930461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.930439 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc07e711-7215-44f0-86b5-69a682869a4d-kube-api-access-rrstw" pod "cc07e711-7215-44f0-86b5-69a682869a4d" (UID: "cc07e711-7215-44f0-86b5-69a682869a4d"). InnerVolumeSpecName "kube-api-access-rrstw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:30.930461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.930441 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-metrics-certs" pod "cc07e711-7215-44f0-86b5-69a682869a4d" (UID: "cc07e711-7215-44f0-86b5-69a682869a4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:30.930461 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:30.930454 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-maas-api-tls" pod "cc07e711-7215-44f0-86b5-69a682869a4d" (UID: "cc07e711-7215-44f0-86b5-69a682869a4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:31.029417 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.029341 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:31.029417 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.029374 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc07e711-7215-44f0-86b5-69a682869a4d-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:31.029417 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.029387 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrstw\" (UniqueName: \"kubernetes.io/projected/cc07e711-7215-44f0-86b5-69a682869a4d-kube-api-access-rrstw\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:31.140460 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.140411 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.315085 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.315011 5931 generic.go:356] "Generic (PLEG): container finished" podID="cc07e711-7215-44f0-86b5-69a682869a4d" containerID="6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b" exitCode=0 Nov 03 22:22:31.315085 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.315060 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" Nov 03 22:22:31.315330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.315094 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" event={"ID":"cc07e711-7215-44f0-86b5-69a682869a4d","Type":"ContainerDied","Data":"6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b"} Nov 03 22:22:31.315330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.315163 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t" event={"ID":"cc07e711-7215-44f0-86b5-69a682869a4d","Type":"ContainerDied","Data":"5ff083f8987a44dbfa4b0029ccac6519c5468979e2b03776a7c03d2f6854cbf8"} Nov 03 22:22:31.315330 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.315195 5931 scope.go:122] "RemoveContainer" containerID="6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b" Nov 03 22:22:31.325019 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.324999 5931 scope.go:122] "RemoveContainer" containerID="6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b" Nov 03 22:22:31.325298 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:31.325276 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b\": container with ID starting with 6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b not found: ID does not exist" containerID="6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b" Nov 03 22:22:31.325377 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.325307 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b"} err="failed to get container status \"6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b\": rpc error: code = NotFound desc = could not find container \"6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b\": container with ID starting with 6d7186c15f728e82f2d064d8e2548f79e7bba71cbe886590c8262ce9548cec8b not found: ID does not exist" Nov 03 22:22:31.480001 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.479972 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t"] Nov 03 22:22:31.499854 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.499814 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.529046 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:31.529026 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddbba-666c9f8dc9-kfd8t"] Nov 03 22:22:32.144660 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:32.144559 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:32.373539 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:32.373501 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:32.508464 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:32.508392 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc07e711-7215-44f0-86b5-69a682869a4d" path="/var/lib/kubelet/pods/cc07e711-7215-44f0-86b5-69a682869a4d/volumes" Nov 03 22:22:32.749829 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:32.749798 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:33.070342 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:33.070308 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:22:33.121581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:33.121535 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:33.490570 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:33.490540 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:34.496117 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:34.496087 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh"] Nov 03 22:22:34.608996 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:34.608967 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd"] Nov 03 22:22:34.972721 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:34.972683 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:35.191302 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:35.191275 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.470157 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.470126 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58"] Nov 03 22:22:37.470592 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.470314 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" podUID="8a6b97c4-9953-4b74-acbe-01ecf1043be4" containerName="maas-api" containerID="cri-o://bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3" gracePeriod=30 Nov 03 22:22:37.663590 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.663560 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.726341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.726286 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:22:37.893621 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.893579 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6b97c4-9953-4b74-acbe-01ecf1043be4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b97c4-9953-4b74-acbe-01ecf1043be4-openshift-service-ca\") pod \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " Nov 03 22:22:37.893778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.893631 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6b97c4-9953-4b74-acbe-01ecf1043be4-kube-api-access-v6x9k\" (UniqueName: \"kubernetes.io/projected/8a6b97c4-9953-4b74-acbe-01ecf1043be4-kube-api-access-v6x9k\") pod \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " Nov 03 22:22:37.893778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.893666 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls\") pod \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " Nov 03 22:22:37.893778 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.893714 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-metrics-certs\") pod \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\" (UID: \"8a6b97c4-9953-4b74-acbe-01ecf1043be4\") " Nov 03 22:22:37.893972 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.893897 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6b97c4-9953-4b74-acbe-01ecf1043be4-openshift-service-ca" pod "8a6b97c4-9953-4b74-acbe-01ecf1043be4" (UID: "8a6b97c4-9953-4b74-acbe-01ecf1043be4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:37.894148 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.894128 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b97c4-9953-4b74-acbe-01ecf1043be4-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:37.895853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.895827 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls" pod "8a6b97c4-9953-4b74-acbe-01ecf1043be4" (UID: "8a6b97c4-9953-4b74-acbe-01ecf1043be4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:37.895974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.895914 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6b97c4-9953-4b74-acbe-01ecf1043be4-kube-api-access-v6x9k" pod "8a6b97c4-9953-4b74-acbe-01ecf1043be4" (UID: "8a6b97c4-9953-4b74-acbe-01ecf1043be4"). InnerVolumeSpecName "kube-api-access-v6x9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:37.895974 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.895922 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-metrics-certs" pod "8a6b97c4-9953-4b74-acbe-01ecf1043be4" (UID: "8a6b97c4-9953-4b74-acbe-01ecf1043be4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:37.994868 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.994814 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6x9k\" (UniqueName: \"kubernetes.io/projected/8a6b97c4-9953-4b74-acbe-01ecf1043be4-kube-api-access-v6x9k\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:37.994868 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.994837 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:37.994868 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:37.994847 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6b97c4-9953-4b74-acbe-01ecf1043be4-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:38.029773 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.029749 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:38.307644 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.307507 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:38.358828 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.358794 5931 generic.go:356] "Generic (PLEG): container finished" podID="8a6b97c4-9953-4b74-acbe-01ecf1043be4" containerID="bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3" exitCode=0 Nov 03 22:22:38.359005 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.358859 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" Nov 03 22:22:38.359005 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.358885 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" event={"ID":"8a6b97c4-9953-4b74-acbe-01ecf1043be4","Type":"ContainerDied","Data":"bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3"} Nov 03 22:22:38.359005 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.358948 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58" event={"ID":"8a6b97c4-9953-4b74-acbe-01ecf1043be4","Type":"ContainerDied","Data":"4aa556246dbea0bc3137c0c9d5e5474ccc10d61a4742493ed5163896ac60a7b9"} Nov 03 22:22:38.359005 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.358971 5931 scope.go:122] "RemoveContainer" containerID="bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3" Nov 03 22:22:38.368924 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.368907 5931 scope.go:122] "RemoveContainer" containerID="bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3" Nov 03 22:22:38.369181 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:38.369158 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3\": container with ID starting with bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3 not found: ID does not exist" containerID="bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3" Nov 03 22:22:38.369256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.369189 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3"} err="failed to get container status \"bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3\": rpc error: code = NotFound desc = could not find container \"bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3\": container with ID starting with bd0e727084708c59b9385af0b153c149786ab9f818e2ce6ae5d5f891af43c8a3 not found: ID does not exist" Nov 03 22:22:38.524163 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.524128 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58"] Nov 03 22:22:38.541977 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.541912 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:38.571812 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:38.571739 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e15bbc-c5f58745b-m9d58"] Nov 03 22:22:39.318571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.318534 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:39.497265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.497235 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" podUID="2a656523-f835-437b-80d2-fff51c87f061" containerName="payload-processing" containerID="cri-o://ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58" gracePeriod=25 Nov 03 22:22:39.551217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.551185 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:39.609842 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.609816 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" podUID="6f745fb4-1903-4e95-b3c9-92429f5400d5" containerName="payload-pre-processing" containerID="cri-o://00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a" gracePeriod=25 Nov 03 22:22:39.748081 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.748057 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 22:22:39.784039 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.784013 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:39.919251 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.919168 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a656523-f835-437b-80d2-fff51c87f061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a656523-f835-437b-80d2-fff51c87f061-plugins-config-volume\") pod \"2a656523-f835-437b-80d2-fff51c87f061\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " Nov 03 22:22:39.919458 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.919338 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a656523-f835-437b-80d2-fff51c87f061-kube-api-access-kl6dh\" (UniqueName: \"kubernetes.io/projected/2a656523-f835-437b-80d2-fff51c87f061-kube-api-access-kl6dh\") pod \"2a656523-f835-437b-80d2-fff51c87f061\" (UID: \"2a656523-f835-437b-80d2-fff51c87f061\") " Nov 03 22:22:39.920766 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.920736 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a656523-f835-437b-80d2-fff51c87f061-plugins-config-volume" pod "2a656523-f835-437b-80d2-fff51c87f061" (UID: "2a656523-f835-437b-80d2-fff51c87f061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:39.922375 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:39.922348 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a656523-f835-437b-80d2-fff51c87f061-kube-api-access-kl6dh" pod "2a656523-f835-437b-80d2-fff51c87f061" (UID: "2a656523-f835-437b-80d2-fff51c87f061"). InnerVolumeSpecName "kube-api-access-kl6dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:40.020629 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.020572 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a656523-f835-437b-80d2-fff51c87f061-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:40.020799 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.020648 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl6dh\" (UniqueName: \"kubernetes.io/projected/2a656523-f835-437b-80d2-fff51c87f061-kube-api-access-kl6dh\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:40.038558 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.038527 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:40.267156 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.267087 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:40.372496 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.372466 5931 generic.go:356] "Generic (PLEG): container finished" podID="2a656523-f835-437b-80d2-fff51c87f061" containerID="ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58" exitCode=0 Nov 03 22:22:40.372652 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.372513 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" Nov 03 22:22:40.372652 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.372509 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" event={"ID":"2a656523-f835-437b-80d2-fff51c87f061","Type":"ContainerDied","Data":"ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58"} Nov 03 22:22:40.372752 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.372652 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh" event={"ID":"2a656523-f835-437b-80d2-fff51c87f061","Type":"ContainerDied","Data":"852292dfab586a264aeb4212b758ab2a223f8809b38bc32adf369966b79f48e3"} Nov 03 22:22:40.372752 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.372677 5931 scope.go:122] "RemoveContainer" containerID="ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58" Nov 03 22:22:40.381841 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.381825 5931 scope.go:122] "RemoveContainer" containerID="ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58" Nov 03 22:22:40.382075 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:40.382056 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58\": container with ID starting with ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58 not found: ID does not exist" containerID="ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58" Nov 03 22:22:40.382145 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.382087 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58"} err="failed to get container status \"ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58\": rpc error: code = NotFound desc = could not find container \"ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58\": container with ID starting with ea853fbe19234e72a0874c622acd5538bacdd836e8756010bae296e7bb87ae58 not found: ID does not exist" Nov 03 22:22:40.505083 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.505064 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:40.505467 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.505449 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6b97c4-9953-4b74-acbe-01ecf1043be4" path="/var/lib/kubelet/pods/8a6b97c4-9953-4b74-acbe-01ecf1043be4/volumes" Nov 03 22:22:40.537495 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.537441 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh"] Nov 03 22:22:40.586745 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:40.586723 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7ee809-68b9f758d6-fz6wh"] Nov 03 22:22:41.861192 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:41.861158 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" containerID="cri-o://b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08" gracePeriod=45 Nov 03 22:22:42.109190 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.109168 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 22:22:42.137851 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.137804 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005066fa-ff22-419b-af08-6f93eb95ce5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/005066fa-ff22-419b-af08-6f93eb95ce5c-tls-certs\") pod \"005066fa-ff22-419b-af08-6f93eb95ce5c\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " Nov 03 22:22:42.137851 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.137847 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-model-cache\") pod \"005066fa-ff22-419b-af08-6f93eb95ce5c\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " Nov 03 22:22:42.138016 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.137870 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005066fa-ff22-419b-af08-6f93eb95ce5c-kube-api-access-bd8nz\" (UniqueName: \"kubernetes.io/projected/005066fa-ff22-419b-af08-6f93eb95ce5c-kube-api-access-bd8nz\") pod \"005066fa-ff22-419b-af08-6f93eb95ce5c\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " Nov 03 22:22:42.138016 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.137980 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-tmp-dir\") pod \"005066fa-ff22-419b-af08-6f93eb95ce5c\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " Nov 03 22:22:42.138132 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138062 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-home\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-home\") pod \"005066fa-ff22-419b-af08-6f93eb95ce5c\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " Nov 03 22:22:42.138132 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138108 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-model-cache" pod "005066fa-ff22-419b-af08-6f93eb95ce5c" (UID: "005066fa-ff22-419b-af08-6f93eb95ce5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:42.138132 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138123 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-dshm\") pod \"005066fa-ff22-419b-af08-6f93eb95ce5c\" (UID: \"005066fa-ff22-419b-af08-6f93eb95ce5c\") " Nov 03 22:22:42.138278 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138225 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-tmp-dir" pod "005066fa-ff22-419b-af08-6f93eb95ce5c" (UID: "005066fa-ff22-419b-af08-6f93eb95ce5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:42.138332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138306 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-home" pod "005066fa-ff22-419b-af08-6f93eb95ce5c" (UID: "005066fa-ff22-419b-af08-6f93eb95ce5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:42.138741 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138676 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:42.138741 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138700 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:42.138741 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.138714 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:42.139998 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.139979 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005066fa-ff22-419b-af08-6f93eb95ce5c-tls-certs" pod "005066fa-ff22-419b-af08-6f93eb95ce5c" (UID: "005066fa-ff22-419b-af08-6f93eb95ce5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:42.140112 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.140098 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-dshm" pod "005066fa-ff22-419b-af08-6f93eb95ce5c" (UID: "005066fa-ff22-419b-af08-6f93eb95ce5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:42.140155 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.140106 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005066fa-ff22-419b-af08-6f93eb95ce5c-kube-api-access-bd8nz" pod "005066fa-ff22-419b-af08-6f93eb95ce5c" (UID: "005066fa-ff22-419b-af08-6f93eb95ce5c"). InnerVolumeSpecName "kube-api-access-bd8nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:42.239745 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.239719 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005066fa-ff22-419b-af08-6f93eb95ce5c-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:42.239745 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.239744 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd8nz\" (UniqueName: \"kubernetes.io/projected/005066fa-ff22-419b-af08-6f93eb95ce5c-kube-api-access-bd8nz\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:42.239908 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.239759 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005066fa-ff22-419b-af08-6f93eb95ce5c-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:42.387786 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.387755 5931 generic.go:356] "Generic (PLEG): container finished" podID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerID="b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08" exitCode=0 Nov 03 22:22:42.387994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.387800 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" Nov 03 22:22:42.387994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.387818 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" event={"ID":"005066fa-ff22-419b-af08-6f93eb95ce5c","Type":"ContainerDied","Data":"b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08"} Nov 03 22:22:42.387994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.387854 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq" event={"ID":"005066fa-ff22-419b-af08-6f93eb95ce5c","Type":"ContainerDied","Data":"e7eb432f7940438ad88ed8b2a75642fcc2d141aea1c2094172212aa7fe336f49"} Nov 03 22:22:42.387994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.387875 5931 scope.go:122] "RemoveContainer" containerID="b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08" Nov 03 22:22:42.403702 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.403683 5931 scope.go:122] "RemoveContainer" containerID="b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08" Nov 03 22:22:42.403996 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:42.403976 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08\": container with ID starting with b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08 not found: ID does not exist" containerID="b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08" Nov 03 22:22:42.404076 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.404005 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08"} err="failed to get container status \"b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08\": rpc error: code = NotFound desc = could not find container \"b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08\": container with ID starting with b85609c8ff1e158aa249bf711f3955b79948c25814b58ea6cba157908e03ae08 not found: ID does not exist" Nov 03 22:22:42.510948 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.510913 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a656523-f835-437b-80d2-fff51c87f061" path="/var/lib/kubelet/pods/2a656523-f835-437b-80d2-fff51c87f061/volumes" Nov 03 22:22:42.554636 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.551798 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq"] Nov 03 22:22:42.560570 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:42.560540 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/premium-w0-7ee809-kserve-5455d5c48d-6vvqq"] Nov 03 22:22:44.507013 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:44.506981 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" path="/var/lib/kubelet/pods/005066fa-ff22-419b-af08-6f93eb95ce5c/volumes" Nov 03 22:22:44.616181 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:44.616145 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:45.349319 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:45.349283 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb"] Nov 03 22:22:45.540265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:45.540235 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.358669 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:46.358633 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.593050 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:46.593006 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.610151 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:46.610096 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8"] Nov 03 22:22:46.610291 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:46.610268 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" containerID="cri-o://612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7" gracePeriod=30 Nov 03 22:22:46.854354 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:46.854328 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.995783 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:46.995717 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" podUID="6f745fb4-1903-4e95-b3c9-92429f5400d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:22:47.093701 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:47.093675 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:49.971469 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:49.971437 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:51.869998 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.869970 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:22:51.926322 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926266 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-envoy\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926322 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926308 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-kube-api-access-rhlln\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-kube-api-access-rhlln\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926487 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926372 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-certs\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926487 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926400 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-socket\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926487 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926452 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-credential-socket\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926656 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926515 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-data\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926656 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926560 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/206bb57e-8add-4c82-be28-d47efff96d09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206bb57e-8add-4c82-be28-d47efff96d09-istiod-ca-cert\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926656 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926639 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/206bb57e-8add-4c82-be28-d47efff96d09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206bb57e-8add-4c82-be28-d47efff96d09-istio-podinfo\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926667 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-istio-token\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-istio-token\") pod \"206bb57e-8add-4c82-be28-d47efff96d09\" (UID: \"206bb57e-8add-4c82-be28-d47efff96d09\") " Nov 03 22:22:51.926847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.926668 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-certs" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:51.927253 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.927056 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-credential-socket" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:51.927253 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.927228 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-data" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:51.927384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.927290 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206bb57e-8add-4c82-be28-d47efff96d09-istiod-ca-cert" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:51.927384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.927354 5931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-credential-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:51.927384 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.927379 5931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-data\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:51.927491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.927394 5931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:51.927571 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.927551 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-socket" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:51.928442 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.928425 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-envoy" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:51.928642 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.928599 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-kube-api-access-rhlln" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "kube-api-access-rhlln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:51.929156 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.929139 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-istio-token" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:51.929206 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:51.929144 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/206bb57e-8add-4c82-be28-d47efff96d09-istio-podinfo" pod "206bb57e-8add-4c82-be28-d47efff96d09" (UID: "206bb57e-8add-4c82-be28-d47efff96d09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:22:52.027833 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.027808 5931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-workload-socket\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.027833 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.027830 5931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206bb57e-8add-4c82-be28-d47efff96d09-istiod-ca-cert\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.027967 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.027840 5931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206bb57e-8add-4c82-be28-d47efff96d09-istio-podinfo\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.027967 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.027849 5931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-istio-token\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.027967 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.027857 5931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206bb57e-8add-4c82-be28-d47efff96d09-istio-envoy\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.027967 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.027866 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhlln\" (UniqueName: \"kubernetes.io/projected/206bb57e-8add-4c82-be28-d47efff96d09-kube-api-access-rhlln\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.445146 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.445110 5931 generic.go:356] "Generic (PLEG): container finished" podID="206bb57e-8add-4c82-be28-d47efff96d09" containerID="612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7" exitCode=0 Nov 03 22:22:52.445332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.445162 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" Nov 03 22:22:52.445332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.445171 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" event={"ID":"206bb57e-8add-4c82-be28-d47efff96d09","Type":"ContainerDied","Data":"612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7"} Nov 03 22:22:52.445332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.445224 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8" event={"ID":"206bb57e-8add-4c82-be28-d47efff96d09","Type":"ContainerDied","Data":"fb466785d78799749c802b981ea34d2380699c38a7885370c125393931593135"} Nov 03 22:22:52.445332 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.445252 5931 scope.go:122] "RemoveContainer" containerID="612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7" Nov 03 22:22:52.455319 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.455303 5931 scope.go:122] "RemoveContainer" containerID="1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c" Nov 03 22:22:52.464120 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.464102 5931 scope.go:122] "RemoveContainer" containerID="612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7" Nov 03 22:22:52.464406 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:52.464387 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7\": container with ID starting with 612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7 not found: ID does not exist" containerID="612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7" Nov 03 22:22:52.464477 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.464413 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7"} err="failed to get container status \"612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7\": rpc error: code = NotFound desc = could not find container \"612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7\": container with ID starting with 612c4c26ce45ebe3ced54b0c85e1e92341dd8cfb5b47609f12697f6177f560e7 not found: ID does not exist" Nov 03 22:22:52.464477 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.464427 5931 scope.go:122] "RemoveContainer" containerID="1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c" Nov 03 22:22:52.464685 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:52.464670 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c\": container with ID starting with 1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c not found: ID does not exist" containerID="1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c" Nov 03 22:22:52.464734 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.464689 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c"} err="failed to get container status \"1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c\": rpc error: code = NotFound desc = could not find container \"1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c\": container with ID starting with 1ce8b67f9c1c8a43e5a349248cbeeaf3720bda827061f9bb4fe89916567c902c not found: ID does not exist" Nov 03 22:22:52.607991 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.607966 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8"] Nov 03 22:22:52.611921 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.611898 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e15bbc-openshift-default-69679ffb49-jrdg8"] Nov 03 22:22:52.723496 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.723435 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w"] Nov 03 22:22:52.987270 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:52.987165 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:53.216257 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:53.216200 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:53.446276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:53.446247 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:54.504993 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.504957 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206bb57e-8add-4c82-be28-d47efff96d09" path="/var/lib/kubelet/pods/206bb57e-8add-4c82-be28-d47efff96d09/volumes" Nov 03 22:22:54.511267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.511240 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2"] Nov 03 22:22:54.511512 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.511477 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" podUID="13d47dae-3cc7-4a35-87cb-fde3815aa4f0" containerName="maas-api" containerID="cri-o://a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f" gracePeriod=30 Nov 03 22:22:54.710190 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.710152 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:54.769546 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.769480 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 22:22:54.853792 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.853765 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-kube-api-access-dtljb\" (UniqueName: \"kubernetes.io/projected/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-kube-api-access-dtljb\") pod \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " Nov 03 22:22:54.853980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.853803 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-metrics-certs\") pod \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " Nov 03 22:22:54.853980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.853827 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls\") pod \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " Nov 03 22:22:54.853980 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.853904 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-openshift-service-ca\") pod \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\" (UID: \"13d47dae-3cc7-4a35-87cb-fde3815aa4f0\") " Nov 03 22:22:54.854388 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.854356 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-openshift-service-ca" pod "13d47dae-3cc7-4a35-87cb-fde3815aa4f0" (UID: "13d47dae-3cc7-4a35-87cb-fde3815aa4f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:54.856100 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.856064 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-metrics-certs" pod "13d47dae-3cc7-4a35-87cb-fde3815aa4f0" (UID: "13d47dae-3cc7-4a35-87cb-fde3815aa4f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:54.856100 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.856084 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls" pod "13d47dae-3cc7-4a35-87cb-fde3815aa4f0" (UID: "13d47dae-3cc7-4a35-87cb-fde3815aa4f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:54.856233 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.856200 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-kube-api-access-dtljb" pod "13d47dae-3cc7-4a35-87cb-fde3815aa4f0" (UID: "13d47dae-3cc7-4a35-87cb-fde3815aa4f0"). InnerVolumeSpecName "kube-api-access-dtljb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:54.955584 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.955554 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:54.955584 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.955578 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtljb\" (UniqueName: \"kubernetes.io/projected/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-kube-api-access-dtljb\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:54.955584 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.955588 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:54.955847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:54.955599 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d47dae-3cc7-4a35-87cb-fde3815aa4f0-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:55.463454 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.463417 5931 generic.go:356] "Generic (PLEG): container finished" podID="13d47dae-3cc7-4a35-87cb-fde3815aa4f0" containerID="a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f" exitCode=0 Nov 03 22:22:55.463670 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.463458 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" Nov 03 22:22:55.463670 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.463503 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" event={"ID":"13d47dae-3cc7-4a35-87cb-fde3815aa4f0","Type":"ContainerDied","Data":"a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f"} Nov 03 22:22:55.463670 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.463553 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2" event={"ID":"13d47dae-3cc7-4a35-87cb-fde3815aa4f0","Type":"ContainerDied","Data":"432dcbe28215cda9a7078ac8b80ea1b0ad3470e558cd25e5dd2d80d294b69313"} Nov 03 22:22:55.463670 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.463577 5931 scope.go:122] "RemoveContainer" containerID="a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f" Nov 03 22:22:55.474891 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.474874 5931 scope.go:122] "RemoveContainer" containerID="a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f" Nov 03 22:22:55.475170 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:22:55.475151 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f\": container with ID starting with a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f not found: ID does not exist" containerID="a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f" Nov 03 22:22:55.475237 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.475180 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f"} err="failed to get container status \"a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f\": rpc error: code = NotFound desc = could not find container \"a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f\": container with ID starting with a79db23c7e43260fbfa09b50d4a6ba8d88cb025c1633a2561037ee991dd78c3f not found: ID does not exist" Nov 03 22:22:55.626230 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.626201 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2"] Nov 03 22:22:55.675342 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:55.675306 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6de8e6-669fdcbd88-bqhf2"] Nov 03 22:22:56.474872 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:56.474842 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:56.504762 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:56.504737 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d47dae-3cc7-4a35-87cb-fde3815aa4f0" path="/var/lib/kubelet/pods/13d47dae-3cc7-4a35-87cb-fde3815aa4f0/volumes" Nov 03 22:22:56.717797 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:56.717771 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:56.995647 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:56.995594 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" podUID="6f745fb4-1903-4e95-b3c9-92429f5400d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:22:57.288603 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:57.288502 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:57.724312 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:57.724277 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" podUID="549c57f9-0b18-48da-9712-73248d9d5cb8" containerName="payload-pre-processing" containerID="cri-o://b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e" gracePeriod=25 Nov 03 22:22:58.942359 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:58.942325 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" podUID="549c57f9-0b18-48da-9712-73248d9d5cb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:22:58.993060 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:58.993022 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk"] Nov 03 22:22:59.021344 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.021315 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:59.100740 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.100711 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx"] Nov 03 22:22:59.408764 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.408730 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:59.629675 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.629650 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:59.856445 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.856423 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 22:22:59.901137 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.901110 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f745fb4-1903-4e95-b3c9-92429f5400d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f745fb4-1903-4e95-b3c9-92429f5400d5-plugins-config-volume\") pod \"6f745fb4-1903-4e95-b3c9-92429f5400d5\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " Nov 03 22:22:59.901264 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.901154 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f745fb4-1903-4e95-b3c9-92429f5400d5-kube-api-access-sqr8f\" (UniqueName: \"kubernetes.io/projected/6f745fb4-1903-4e95-b3c9-92429f5400d5-kube-api-access-sqr8f\") pod \"6f745fb4-1903-4e95-b3c9-92429f5400d5\" (UID: \"6f745fb4-1903-4e95-b3c9-92429f5400d5\") " Nov 03 22:22:59.901436 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.901414 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f745fb4-1903-4e95-b3c9-92429f5400d5-plugins-config-volume" pod "6f745fb4-1903-4e95-b3c9-92429f5400d5" (UID: "6f745fb4-1903-4e95-b3c9-92429f5400d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:59.901558 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.901545 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f745fb4-1903-4e95-b3c9-92429f5400d5-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:59.903209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:22:59.903190 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f745fb4-1903-4e95-b3c9-92429f5400d5-kube-api-access-sqr8f" pod "6f745fb4-1903-4e95-b3c9-92429f5400d5" (UID: "6f745fb4-1903-4e95-b3c9-92429f5400d5"). InnerVolumeSpecName "kube-api-access-sqr8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:00.002538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.002483 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqr8f\" (UniqueName: \"kubernetes.io/projected/6f745fb4-1903-4e95-b3c9-92429f5400d5-kube-api-access-sqr8f\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:00.369532 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.369504 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" podUID="c30da697-3933-4894-8f70-da325802d8b6" containerName="main" containerID="cri-o://145d6829826c48cc42b13ee18cefc850588e63cfca9da447d3b697070b4e1c06" gracePeriod=45 Nov 03 22:23:00.373786 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.373758 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" podUID="c30da697-3933-4894-8f70-da325802d8b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": EOF" Nov 03 22:23:00.496968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.496934 5931 generic.go:356] "Generic (PLEG): container finished" podID="c30da697-3933-4894-8f70-da325802d8b6" containerID="145d6829826c48cc42b13ee18cefc850588e63cfca9da447d3b697070b4e1c06" exitCode=0 Nov 03 22:23:00.497141 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.497003 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" event={"ID":"c30da697-3933-4894-8f70-da325802d8b6","Type":"ContainerDied","Data":"145d6829826c48cc42b13ee18cefc850588e63cfca9da447d3b697070b4e1c06"} Nov 03 22:23:00.498792 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.498762 5931 generic.go:356] "Generic (PLEG): container finished" podID="6f745fb4-1903-4e95-b3c9-92429f5400d5" containerID="00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a" exitCode=0 Nov 03 22:23:00.498903 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.498863 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" Nov 03 22:23:00.504491 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.504467 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" event={"ID":"6f745fb4-1903-4e95-b3c9-92429f5400d5","Type":"ContainerDied","Data":"00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a"} Nov 03 22:23:00.504599 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.504508 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd" event={"ID":"6f745fb4-1903-4e95-b3c9-92429f5400d5","Type":"ContainerDied","Data":"37b896a07c554dcab1d4c373d61fe78794a5e9a92a0831a400c02fc4aa27fb9f"} Nov 03 22:23:00.504599 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.504539 5931 scope.go:122] "RemoveContainer" containerID="00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a" Nov 03 22:23:00.514537 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.514507 5931 scope.go:122] "RemoveContainer" containerID="00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a" Nov 03 22:23:00.514877 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:00.514858 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a\": container with ID starting with 00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a not found: ID does not exist" containerID="00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a" Nov 03 22:23:00.514955 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.514886 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a"} err="failed to get container status \"00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a\": rpc error: code = NotFound desc = could not find container \"00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a\": container with ID starting with 00cd9d1e32fe9cd43f88249db76c71b520b43fa8e9f03c3dc18b4a617ca8c14a not found: ID does not exist" Nov 03 22:23:00.610318 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.610296 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 22:23:00.661414 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.661377 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd"] Nov 03 22:23:00.664979 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.664956 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7ee809-56d555bb47-6p4fd"] Nov 03 22:23:00.709372 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709344 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-tmp-dir\") pod \"c30da697-3933-4894-8f70-da325802d8b6\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " Nov 03 22:23:00.709508 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709387 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30da697-3933-4894-8f70-da325802d8b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c30da697-3933-4894-8f70-da325802d8b6-tls-certs\") pod \"c30da697-3933-4894-8f70-da325802d8b6\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " Nov 03 22:23:00.709508 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709415 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30da697-3933-4894-8f70-da325802d8b6-kube-api-access-cwtql\" (UniqueName: \"kubernetes.io/projected/c30da697-3933-4894-8f70-da325802d8b6-kube-api-access-cwtql\") pod \"c30da697-3933-4894-8f70-da325802d8b6\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " Nov 03 22:23:00.709508 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709445 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-home\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-home\") pod \"c30da697-3933-4894-8f70-da325802d8b6\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " Nov 03 22:23:00.709717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709544 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-dshm\") pod \"c30da697-3933-4894-8f70-da325802d8b6\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " Nov 03 22:23:00.709717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709559 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-tmp-dir" pod "c30da697-3933-4894-8f70-da325802d8b6" (UID: "c30da697-3933-4894-8f70-da325802d8b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:00.709717 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709566 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-model-cache\") pod \"c30da697-3933-4894-8f70-da325802d8b6\" (UID: \"c30da697-3933-4894-8f70-da325802d8b6\") " Nov 03 22:23:00.709885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709805 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-model-cache" pod "c30da697-3933-4894-8f70-da325802d8b6" (UID: "c30da697-3933-4894-8f70-da325802d8b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:00.709885 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.709817 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-home" pod "c30da697-3933-4894-8f70-da325802d8b6" (UID: "c30da697-3933-4894-8f70-da325802d8b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:00.710023 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.710007 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:00.710085 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.710042 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:00.710085 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.710057 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:00.711973 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.711952 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-dshm" pod "c30da697-3933-4894-8f70-da325802d8b6" (UID: "c30da697-3933-4894-8f70-da325802d8b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:00.712151 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.712132 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30da697-3933-4894-8f70-da325802d8b6-tls-certs" pod "c30da697-3933-4894-8f70-da325802d8b6" (UID: "c30da697-3933-4894-8f70-da325802d8b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:00.712219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.712154 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30da697-3933-4894-8f70-da325802d8b6-kube-api-access-cwtql" pod "c30da697-3933-4894-8f70-da325802d8b6" (UID: "c30da697-3933-4894-8f70-da325802d8b6"). InnerVolumeSpecName "kube-api-access-cwtql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:00.810779 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.810749 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30da697-3933-4894-8f70-da325802d8b6-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:00.810779 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.810779 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30da697-3933-4894-8f70-da325802d8b6-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:00.810952 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:00.810791 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtql\" (UniqueName: \"kubernetes.io/projected/c30da697-3933-4894-8f70-da325802d8b6-kube-api-access-cwtql\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:01.508410 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:01.508388 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" Nov 03 22:23:01.508921 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:01.508438 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb" event={"ID":"c30da697-3933-4894-8f70-da325802d8b6","Type":"ContainerDied","Data":"6095fcd70c11c0656062cc0437c7dbd6f81342a56bb3544e76a21c355539d499"} Nov 03 22:23:01.508921 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:01.508494 5931 scope.go:122] "RemoveContainer" containerID="145d6829826c48cc42b13ee18cefc850588e63cfca9da447d3b697070b4e1c06" Nov 03 22:23:01.670287 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:01.670262 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb"] Nov 03 22:23:01.719583 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:01.719556 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-2-w2-d71a74-kserve-6dc95867f6-lvrcb"] Nov 03 22:23:01.919353 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:01.919324 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:02.505928 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:02.505895 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f745fb4-1903-4e95-b3c9-92429f5400d5" path="/var/lib/kubelet/pods/6f745fb4-1903-4e95-b3c9-92429f5400d5/volumes" Nov 03 22:23:02.506276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:02.506262 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30da697-3933-4894-8f70-da325802d8b6" path="/var/lib/kubelet/pods/c30da697-3933-4894-8f70-da325802d8b6/volumes" Nov 03 22:23:03.067511 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:03.067474 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:23:03.073946 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:03.073919 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:03.310876 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:03.310840 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:03.993319 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:03.993269 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" podUID="bd483f46-2923-4f02-bfe1-6fe9ad32d374" containerName="payload-processing" containerID="cri-o://26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b" gracePeriod=25 Nov 03 22:23:04.101382 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.101309 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" podUID="ecc6892b-cda8-479d-9c04-b5d16efd8b4a" containerName="payload-pre-processing" containerID="cri-o://dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036" gracePeriod=25 Nov 03 22:23:04.216690 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.216648 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:04.271162 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.271139 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:23:04.347183 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.347157 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd483f46-2923-4f02-bfe1-6fe9ad32d374-kube-api-access-zcndw\" (UniqueName: \"kubernetes.io/projected/bd483f46-2923-4f02-bfe1-6fe9ad32d374-kube-api-access-zcndw\") pod \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " Nov 03 22:23:04.347364 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.347229 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd483f46-2923-4f02-bfe1-6fe9ad32d374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd483f46-2923-4f02-bfe1-6fe9ad32d374-plugins-config-volume\") pod \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\" (UID: \"bd483f46-2923-4f02-bfe1-6fe9ad32d374\") " Nov 03 22:23:04.347585 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.347563 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd483f46-2923-4f02-bfe1-6fe9ad32d374-plugins-config-volume" pod "bd483f46-2923-4f02-bfe1-6fe9ad32d374" (UID: "bd483f46-2923-4f02-bfe1-6fe9ad32d374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:04.349267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.349240 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd483f46-2923-4f02-bfe1-6fe9ad32d374-kube-api-access-zcndw" pod "bd483f46-2923-4f02-bfe1-6fe9ad32d374" (UID: "bd483f46-2923-4f02-bfe1-6fe9ad32d374"). InnerVolumeSpecName "kube-api-access-zcndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:04.363910 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.363889 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:23:04.441631 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.441592 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:04.448725 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.448705 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-plugins-config-volume\") pod \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " Nov 03 22:23:04.448893 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.448877 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-kube-api-access-hglvn\" (UniqueName: \"kubernetes.io/projected/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-kube-api-access-hglvn\") pod \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\" (UID: \"ecc6892b-cda8-479d-9c04-b5d16efd8b4a\") " Nov 03 22:23:04.449101 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.449081 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-plugins-config-volume" pod "ecc6892b-cda8-479d-9c04-b5d16efd8b4a" (UID: "ecc6892b-cda8-479d-9c04-b5d16efd8b4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:04.449205 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.449191 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcndw\" (UniqueName: \"kubernetes.io/projected/bd483f46-2923-4f02-bfe1-6fe9ad32d374-kube-api-access-zcndw\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:04.449241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.449208 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:04.449241 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.449217 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd483f46-2923-4f02-bfe1-6fe9ad32d374-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:04.450935 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.450913 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-kube-api-access-hglvn" pod "ecc6892b-cda8-479d-9c04-b5d16efd8b4a" (UID: "ecc6892b-cda8-479d-9c04-b5d16efd8b4a"). InnerVolumeSpecName "kube-api-access-hglvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:04.527551 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.527526 5931 generic.go:356] "Generic (PLEG): container finished" podID="ecc6892b-cda8-479d-9c04-b5d16efd8b4a" containerID="dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036" exitCode=0 Nov 03 22:23:04.527687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.527577 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" Nov 03 22:23:04.527687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.527627 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" event={"ID":"ecc6892b-cda8-479d-9c04-b5d16efd8b4a","Type":"ContainerDied","Data":"dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036"} Nov 03 22:23:04.527687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.527674 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx" event={"ID":"ecc6892b-cda8-479d-9c04-b5d16efd8b4a","Type":"ContainerDied","Data":"55773835b71889103b62ad15bb8d7bb7d6eb3873260857dd3719ba5d4096a6fa"} Nov 03 22:23:04.527844 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.527700 5931 scope.go:122] "RemoveContainer" containerID="dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036" Nov 03 22:23:04.529582 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.529546 5931 generic.go:356] "Generic (PLEG): container finished" podID="bd483f46-2923-4f02-bfe1-6fe9ad32d374" containerID="26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b" exitCode=0 Nov 03 22:23:04.529582 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.529591 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" event={"ID":"bd483f46-2923-4f02-bfe1-6fe9ad32d374","Type":"ContainerDied","Data":"26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b"} Nov 03 22:23:04.529815 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.529660 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" event={"ID":"bd483f46-2923-4f02-bfe1-6fe9ad32d374","Type":"ContainerDied","Data":"4f43648cc5a611b23ba7381a55b063f2c41ca9b4b3c950ec1d7831a01dc1aa74"} Nov 03 22:23:04.529815 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.529725 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk" Nov 03 22:23:04.538133 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.538116 5931 scope.go:122] "RemoveContainer" containerID="dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036" Nov 03 22:23:04.538356 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:04.538339 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036\": container with ID starting with dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036 not found: ID does not exist" containerID="dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036" Nov 03 22:23:04.538405 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.538364 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036"} err="failed to get container status \"dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036\": rpc error: code = NotFound desc = could not find container \"dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036\": container with ID starting with dcf20dec28972b572a5c56ff6bc9a9b01e1a907c693f55a4ca0f5c9f097b8036 not found: ID does not exist" Nov 03 22:23:04.538405 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.538379 5931 scope.go:122] "RemoveContainer" containerID="26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b" Nov 03 22:23:04.546281 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.546265 5931 scope.go:122] "RemoveContainer" containerID="26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b" Nov 03 22:23:04.546530 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:04.546514 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b\": container with ID starting with 26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b not found: ID does not exist" containerID="26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b" Nov 03 22:23:04.546581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.546536 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b"} err="failed to get container status \"26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b\": rpc error: code = NotFound desc = could not find container \"26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b\": container with ID starting with 26d1ab707c2915ee426b05af5618e0b97b2ad317226e8a4400ae03020ddef99b not found: ID does not exist" Nov 03 22:23:04.549996 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.549975 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hglvn\" (UniqueName: \"kubernetes.io/projected/ecc6892b-cda8-479d-9c04-b5d16efd8b4a-kube-api-access-hglvn\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:04.668963 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.668941 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:04.689120 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.689099 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk"] Nov 03 22:23:04.692797 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.692780 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fb477-75887645b6-gglkk"] Nov 03 22:23:04.799392 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.799345 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx"] Nov 03 22:23:04.805636 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:04.805604 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fb477-6678f79cb7-vz2tx"] Nov 03 22:23:06.203780 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:06.203751 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:06.504831 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:06.504756 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd483f46-2923-4f02-bfe1-6fe9ad32d374" path="/var/lib/kubelet/pods/bd483f46-2923-4f02-bfe1-6fe9ad32d374/volumes" Nov 03 22:23:06.505134 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:06.505120 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc6892b-cda8-479d-9c04-b5d16efd8b4a" path="/var/lib/kubelet/pods/ecc6892b-cda8-479d-9c04-b5d16efd8b4a/volumes" Nov 03 22:23:07.397476 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.397445 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss"] Nov 03 22:23:07.397925 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.397655 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" podUID="6ae84c73-222d-40e4-ba77-e046ef07b02f" containerName="maas-api" containerID="cri-o://dd017ee18931c435a053135bd42cbcb57664d48cdf518d9864e97619dc872b3f" gracePeriod=30 Nov 03 22:23:07.549351 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.549322 5931 generic.go:356] "Generic (PLEG): container finished" podID="6ae84c73-222d-40e4-ba77-e046ef07b02f" containerID="dd017ee18931c435a053135bd42cbcb57664d48cdf518d9864e97619dc872b3f" exitCode=0 Nov 03 22:23:07.549687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.549375 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" event={"ID":"6ae84c73-222d-40e4-ba77-e046ef07b02f","Type":"ContainerDied","Data":"dd017ee18931c435a053135bd42cbcb57664d48cdf518d9864e97619dc872b3f"} Nov 03 22:23:07.604947 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.604918 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:07.647220 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.647199 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 22:23:07.776968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.776886 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae84c73-222d-40e4-ba77-e046ef07b02f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae84c73-222d-40e4-ba77-e046ef07b02f-openshift-service-ca\") pod \"6ae84c73-222d-40e4-ba77-e046ef07b02f\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " Nov 03 22:23:07.776968 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.776947 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls\") pod \"6ae84c73-222d-40e4-ba77-e046ef07b02f\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " Nov 03 22:23:07.777169 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.776970 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-metrics-certs\") pod \"6ae84c73-222d-40e4-ba77-e046ef07b02f\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " Nov 03 22:23:07.777169 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.777014 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae84c73-222d-40e4-ba77-e046ef07b02f-kube-api-access-rl6bk\" (UniqueName: \"kubernetes.io/projected/6ae84c73-222d-40e4-ba77-e046ef07b02f-kube-api-access-rl6bk\") pod \"6ae84c73-222d-40e4-ba77-e046ef07b02f\" (UID: \"6ae84c73-222d-40e4-ba77-e046ef07b02f\") " Nov 03 22:23:07.777279 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.777236 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae84c73-222d-40e4-ba77-e046ef07b02f-openshift-service-ca" pod "6ae84c73-222d-40e4-ba77-e046ef07b02f" (UID: "6ae84c73-222d-40e4-ba77-e046ef07b02f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:07.777381 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.777357 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae84c73-222d-40e4-ba77-e046ef07b02f-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:07.779226 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.779200 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls" pod "6ae84c73-222d-40e4-ba77-e046ef07b02f" (UID: "6ae84c73-222d-40e4-ba77-e046ef07b02f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:07.779783 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.779426 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae84c73-222d-40e4-ba77-e046ef07b02f-kube-api-access-rl6bk" pod "6ae84c73-222d-40e4-ba77-e046ef07b02f" (UID: "6ae84c73-222d-40e4-ba77-e046ef07b02f"). InnerVolumeSpecName "kube-api-access-rl6bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:07.779783 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.779515 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-metrics-certs" pod "6ae84c73-222d-40e4-ba77-e046ef07b02f" (UID: "6ae84c73-222d-40e4-ba77-e046ef07b02f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:07.834066 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.834038 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:07.878201 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.878174 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:07.878201 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.878196 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae84c73-222d-40e4-ba77-e046ef07b02f-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:07.878321 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:07.878205 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6bk\" (UniqueName: \"kubernetes.io/projected/6ae84c73-222d-40e4-ba77-e046ef07b02f-kube-api-access-rl6bk\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:08.063024 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.062964 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:08.535764 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:08.535722 5931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae84c73_222d_40e4_ba77_e046ef07b02f.slice/crio-f64d977bc41c0cde17d0f2891f6cac8e3cab609c6431687a0bc08049acbbdc16\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae84c73_222d_40e4_ba77_e046ef07b02f.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:23:08.555415 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.555388 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" Nov 03 22:23:08.555540 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.555390 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss" event={"ID":"6ae84c73-222d-40e4-ba77-e046ef07b02f","Type":"ContainerDied","Data":"f64d977bc41c0cde17d0f2891f6cac8e3cab609c6431687a0bc08049acbbdc16"} Nov 03 22:23:08.555583 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.555539 5931 scope.go:122] "RemoveContainer" containerID="dd017ee18931c435a053135bd42cbcb57664d48cdf518d9864e97619dc872b3f" Nov 03 22:23:08.717852 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.717824 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss"] Nov 03 22:23:08.754900 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.754837 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:08.767014 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.766993 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7ee809-746f5dd89d-rcgss"] Nov 03 22:23:08.942329 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:08.942297 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" podUID="549c57f9-0b18-48da-9712-73248d9d5cb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:23:09.209482 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:09.209378 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:09.483209 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:09.483106 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:09.741276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:09.741209 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:10.183195 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:10.183169 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:10.504602 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:10.504525 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae84c73-222d-40e4-ba77-e046ef07b02f" path="/var/lib/kubelet/pods/6ae84c73-222d-40e4-ba77-e046ef07b02f/volumes" Nov 03 22:23:10.514502 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:10.514480 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:14.741685 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:14.741551 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:14.981284 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:14.981255 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:15.275487 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:15.275452 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:15.660252 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:15.660223 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:16.351219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:16.351187 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:16.602705 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:16.602632 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:16.862219 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:16.862130 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:17.975519 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:17.975495 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 22:23:18.063572 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.063495 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549c57f9-0b18-48da-9712-73248d9d5cb8-kube-api-access-jsnwn\" (UniqueName: \"kubernetes.io/projected/549c57f9-0b18-48da-9712-73248d9d5cb8-kube-api-access-jsnwn\") pod \"549c57f9-0b18-48da-9712-73248d9d5cb8\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " Nov 03 22:23:18.063572 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.063543 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549c57f9-0b18-48da-9712-73248d9d5cb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549c57f9-0b18-48da-9712-73248d9d5cb8-plugins-config-volume\") pod \"549c57f9-0b18-48da-9712-73248d9d5cb8\" (UID: \"549c57f9-0b18-48da-9712-73248d9d5cb8\") " Nov 03 22:23:18.063946 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.063926 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549c57f9-0b18-48da-9712-73248d9d5cb8-plugins-config-volume" pod "549c57f9-0b18-48da-9712-73248d9d5cb8" (UID: "549c57f9-0b18-48da-9712-73248d9d5cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:18.065538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.065513 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549c57f9-0b18-48da-9712-73248d9d5cb8-kube-api-access-jsnwn" pod "549c57f9-0b18-48da-9712-73248d9d5cb8" (UID: "549c57f9-0b18-48da-9712-73248d9d5cb8"). InnerVolumeSpecName "kube-api-access-jsnwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:18.164341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.164298 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsnwn\" (UniqueName: \"kubernetes.io/projected/549c57f9-0b18-48da-9712-73248d9d5cb8-kube-api-access-jsnwn\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:18.164341 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.164324 5931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549c57f9-0b18-48da-9712-73248d9d5cb8-plugins-config-volume\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:18.611345 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.611310 5931 generic.go:356] "Generic (PLEG): container finished" podID="549c57f9-0b18-48da-9712-73248d9d5cb8" containerID="b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e" exitCode=0 Nov 03 22:23:18.611503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.611356 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" Nov 03 22:23:18.611503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.611375 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" event={"ID":"549c57f9-0b18-48da-9712-73248d9d5cb8","Type":"ContainerDied","Data":"b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e"} Nov 03 22:23:18.611503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.611428 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w" event={"ID":"549c57f9-0b18-48da-9712-73248d9d5cb8","Type":"ContainerDied","Data":"99bd743a925e525157f4364462f69f3e3829fe198834242262278a4bfae2adb5"} Nov 03 22:23:18.611503 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.611450 5931 scope.go:122] "RemoveContainer" containerID="b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e" Nov 03 22:23:18.621488 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.621470 5931 scope.go:122] "RemoveContainer" containerID="b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e" Nov 03 22:23:18.621768 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:18.621743 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e\": container with ID starting with b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e not found: ID does not exist" containerID="b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e" Nov 03 22:23:18.621854 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.621775 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e"} err="failed to get container status \"b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e\": rpc error: code = NotFound desc = could not find container \"b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e\": container with ID starting with b5b062444db2f3a56ee4552417cb5ff07c56bcd37c0b4a74cc59c22f313c690e not found: ID does not exist" Nov 03 22:23:18.774265 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.774192 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w"] Nov 03 22:23:18.822419 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:18.822376 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d71a74-cd7c57b47-f4w6w"] Nov 03 22:23:19.169579 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.169536 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct"] Nov 03 22:23:19.170112 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.169753 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" podUID="f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" containerName="maas-api" containerID="cri-o://6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c" gracePeriod=30 Nov 03 22:23:19.364429 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.362476 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:19.420657 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.420593 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 22:23:19.579401 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.579369 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-kube-api-access-p7nz2\" (UniqueName: \"kubernetes.io/projected/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-kube-api-access-p7nz2\") pod \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " Nov 03 22:23:19.579594 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.579457 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-metrics-certs\") pod \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " Nov 03 22:23:19.579594 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.579556 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls\") pod \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " Nov 03 22:23:19.579775 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.579639 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-openshift-service-ca\") pod \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\" (UID: \"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997\") " Nov 03 22:23:19.580063 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.580033 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-openshift-service-ca" pod "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" (UID: "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:19.581693 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.581669 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-metrics-certs" pod "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" (UID: "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:19.581788 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.581769 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls" pod "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" (UID: "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:19.581864 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.581836 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-kube-api-access-p7nz2" pod "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" (UID: "f95cb44f-4d6a-48d6-a0a2-6e912d5ff997"). InnerVolumeSpecName "kube-api-access-p7nz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:19.621348 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.621319 5931 generic.go:356] "Generic (PLEG): container finished" podID="f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" containerID="6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c" exitCode=0 Nov 03 22:23:19.621513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.621390 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" Nov 03 22:23:19.621513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.621406 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" event={"ID":"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997","Type":"ContainerDied","Data":"6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c"} Nov 03 22:23:19.621513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.621461 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct" event={"ID":"f95cb44f-4d6a-48d6-a0a2-6e912d5ff997","Type":"ContainerDied","Data":"a3d741da6350fe904451d70dcc6c64a572c7ac1f669a253b8d4482a30145ad44"} Nov 03 22:23:19.621513 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.621482 5931 scope.go:122] "RemoveContainer" containerID="6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c" Nov 03 22:23:19.637368 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.637352 5931 scope.go:122] "RemoveContainer" containerID="6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c" Nov 03 22:23:19.637727 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:19.637708 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c\": container with ID starting with 6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c not found: ID does not exist" containerID="6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c" Nov 03 22:23:19.637795 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.637736 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c"} err="failed to get container status \"6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c\": rpc error: code = NotFound desc = could not find container \"6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c\": container with ID starting with 6ef302302756243a3f9f2db9aaa2c2d7b2cbe230cb871b9042bcf2568b18b36c not found: ID does not exist" Nov 03 22:23:19.680661 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.680589 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:19.680661 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.680649 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:19.680843 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.680664 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:19.680843 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.680678 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7nz2\" (UniqueName: \"kubernetes.io/projected/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997-kube-api-access-p7nz2\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:19.782721 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.782683 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct"] Nov 03 22:23:19.784730 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:19.784703 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d71a74-76487f55c4-5s5ct"] Nov 03 22:23:20.506316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:20.506277 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549c57f9-0b18-48da-9712-73248d9d5cb8" path="/var/lib/kubelet/pods/549c57f9-0b18-48da-9712-73248d9d5cb8/volumes" Nov 03 22:23:20.506836 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:20.506816 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" path="/var/lib/kubelet/pods/f95cb44f-4d6a-48d6-a0a2-6e912d5ff997/volumes" Nov 03 22:23:20.517553 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:20.517520 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv"] Nov 03 22:23:20.567132 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:20.567098 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd"] Nov 03 22:23:20.772917 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:20.772839 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:21.002845 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:21.002813 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:21.228624 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:21.228567 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:21.464440 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:21.464410 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:21.913366 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:21.913336 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:22.151808 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:22.151779 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:23.350727 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:23.350690 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:25.894848 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:25.894816 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:26.128918 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:26.128879 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:26.349763 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:26.349732 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:28.109377 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:28.109344 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:28.358665 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:28.358635 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:28.602285 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:28.602254 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:31.022054 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:31.022024 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:33.068250 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:33.068211 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:23:35.544672 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.544636 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" containerID="cri-o://3a9453aedbd4dea0a90cd317d552e5f8b79d3fcdd28c8a5de86c1f70f08aec11" gracePeriod=45 Nov 03 22:23:35.587597 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.587566 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" containerID="cri-o://5261b4c25850569605464806bc31688e98acc558a9370a5ea453b9a8cfc519ac" gracePeriod=45 Nov 03 22:23:35.590183 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.590153 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused" Nov 03 22:23:35.591690 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.591660 5931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": EOF" Nov 03 22:23:35.711516 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.711483 5931 generic.go:356] "Generic (PLEG): container finished" podID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerID="3a9453aedbd4dea0a90cd317d552e5f8b79d3fcdd28c8a5de86c1f70f08aec11" exitCode=0 Nov 03 22:23:35.711649 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.711556 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" event={"ID":"858f0ceb-1bfa-4d79-a108-bff88911fdc2","Type":"ContainerDied","Data":"3a9453aedbd4dea0a90cd317d552e5f8b79d3fcdd28c8a5de86c1f70f08aec11"} Nov 03 22:23:35.713362 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.713344 5931 generic.go:356] "Generic (PLEG): container finished" podID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerID="5261b4c25850569605464806bc31688e98acc558a9370a5ea453b9a8cfc519ac" exitCode=0 Nov 03 22:23:35.713449 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.713410 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" event={"ID":"c74eaac2-2350-4ec4-8601-a209a18500ef","Type":"ContainerDied","Data":"5261b4c25850569605464806bc31688e98acc558a9370a5ea453b9a8cfc519ac"} Nov 03 22:23:35.832001 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.831972 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 22:23:35.857600 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.857577 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 22:23:35.931855 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.931819 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74eaac2-2350-4ec4-8601-a209a18500ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/c74eaac2-2350-4ec4-8601-a209a18500ef-tls-certs\") pod \"c74eaac2-2350-4ec4-8601-a209a18500ef\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " Nov 03 22:23:35.932015 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.931888 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tls-certs\") pod \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " Nov 03 22:23:35.932015 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.931922 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858f0ceb-1bfa-4d79-a108-bff88911fdc2-kube-api-access-nwlvf\" (UniqueName: \"kubernetes.io/projected/858f0ceb-1bfa-4d79-a108-bff88911fdc2-kube-api-access-nwlvf\") pod \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " Nov 03 22:23:35.932015 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.931964 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-model-cache\") pod \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " Nov 03 22:23:35.932015 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.931997 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-dshm\") pod \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " Nov 03 22:23:35.932256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932026 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-dshm\") pod \"c74eaac2-2350-4ec4-8601-a209a18500ef\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " Nov 03 22:23:35.932256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932078 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74eaac2-2350-4ec4-8601-a209a18500ef-kube-api-access-dmjxb\" (UniqueName: \"kubernetes.io/projected/c74eaac2-2350-4ec4-8601-a209a18500ef-kube-api-access-dmjxb\") pod \"c74eaac2-2350-4ec4-8601-a209a18500ef\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " Nov 03 22:23:35.932256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932113 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-home\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-home\") pod \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " Nov 03 22:23:35.932256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932148 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-home\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-home\") pod \"c74eaac2-2350-4ec4-8601-a209a18500ef\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " Nov 03 22:23:35.932256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932182 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-tmp-dir\") pod \"c74eaac2-2350-4ec4-8601-a209a18500ef\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " Nov 03 22:23:35.932256 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932245 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tmp-dir\") pod \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\" (UID: \"858f0ceb-1bfa-4d79-a108-bff88911fdc2\") " Nov 03 22:23:35.932604 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932309 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-model-cache\") pod \"c74eaac2-2350-4ec4-8601-a209a18500ef\" (UID: \"c74eaac2-2350-4ec4-8601-a209a18500ef\") " Nov 03 22:23:35.932737 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932679 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-home" pod "c74eaac2-2350-4ec4-8601-a209a18500ef" (UID: "c74eaac2-2350-4ec4-8601-a209a18500ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.932826 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932718 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-model-cache" pod "858f0ceb-1bfa-4d79-a108-bff88911fdc2" (UID: "858f0ceb-1bfa-4d79-a108-bff88911fdc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.932904 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932882 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-model-cache" pod "c74eaac2-2350-4ec4-8601-a209a18500ef" (UID: "c74eaac2-2350-4ec4-8601-a209a18500ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.933129 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932908 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-tmp-dir" pod "c74eaac2-2350-4ec4-8601-a209a18500ef" (UID: "c74eaac2-2350-4ec4-8601-a209a18500ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.933129 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.932920 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-home" pod "858f0ceb-1bfa-4d79-a108-bff88911fdc2" (UID: "858f0ceb-1bfa-4d79-a108-bff88911fdc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.933129 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.933091 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tmp-dir" pod "858f0ceb-1bfa-4d79-a108-bff88911fdc2" (UID: "858f0ceb-1bfa-4d79-a108-bff88911fdc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.934761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.934596 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74eaac2-2350-4ec4-8601-a209a18500ef-tls-certs" pod "c74eaac2-2350-4ec4-8601-a209a18500ef" (UID: "c74eaac2-2350-4ec4-8601-a209a18500ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:35.935198 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.935167 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74eaac2-2350-4ec4-8601-a209a18500ef-kube-api-access-dmjxb" pod "c74eaac2-2350-4ec4-8601-a209a18500ef" (UID: "c74eaac2-2350-4ec4-8601-a209a18500ef"). InnerVolumeSpecName "kube-api-access-dmjxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:35.935198 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.935181 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858f0ceb-1bfa-4d79-a108-bff88911fdc2-kube-api-access-nwlvf" pod "858f0ceb-1bfa-4d79-a108-bff88911fdc2" (UID: "858f0ceb-1bfa-4d79-a108-bff88911fdc2"). InnerVolumeSpecName "kube-api-access-nwlvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:35.935356 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.935308 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-dshm" pod "858f0ceb-1bfa-4d79-a108-bff88911fdc2" (UID: "858f0ceb-1bfa-4d79-a108-bff88911fdc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.935574 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.935551 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-dshm" pod "c74eaac2-2350-4ec4-8601-a209a18500ef" (UID: "c74eaac2-2350-4ec4-8601-a209a18500ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:35.936200 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:35.936182 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tls-certs" pod "858f0ceb-1bfa-4d79-a108-bff88911fdc2" (UID: "858f0ceb-1bfa-4d79-a108-bff88911fdc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:36.033388 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033362 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033388 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033386 5931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-home\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033409 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033418 5931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tmp-dir\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033425 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033435 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74eaac2-2350-4ec4-8601-a209a18500ef-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033443 5931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858f0ceb-1bfa-4d79-a108-bff88911fdc2-tls-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033452 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwlvf\" (UniqueName: \"kubernetes.io/projected/858f0ceb-1bfa-4d79-a108-bff88911fdc2-kube-api-access-nwlvf\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033462 5931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-model-cache\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033470 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858f0ceb-1bfa-4d79-a108-bff88911fdc2-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033478 5931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74eaac2-2350-4ec4-8601-a209a18500ef-dshm\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.033581 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.033486 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmjxb\" (UniqueName: \"kubernetes.io/projected/c74eaac2-2350-4ec4-8601-a209a18500ef-kube-api-access-dmjxb\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:36.719157 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.719128 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" Nov 03 22:23:36.719561 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.719158 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd" event={"ID":"c74eaac2-2350-4ec4-8601-a209a18500ef","Type":"ContainerDied","Data":"5f2fa578cd91333c7e0efb1eddc7bcf5c163812965e8b0404bd169006022fac6"} Nov 03 22:23:36.719561 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.719210 5931 scope.go:122] "RemoveContainer" containerID="5261b4c25850569605464806bc31688e98acc558a9370a5ea453b9a8cfc519ac" Nov 03 22:23:36.720761 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.720736 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" event={"ID":"858f0ceb-1bfa-4d79-a108-bff88911fdc2","Type":"ContainerDied","Data":"f62ce1c533d0dcd0e414eac354bb19990e2fb9c8233df468631dbefc9461bc6c"} Nov 03 22:23:36.720862 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.720825 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv" Nov 03 22:23:36.727853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.727836 5931 scope.go:122] "RemoveContainer" containerID="3a9453aedbd4dea0a90cd317d552e5f8b79d3fcdd28c8a5de86c1f70f08aec11" Nov 03 22:23:36.880220 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.880192 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd"] Nov 03 22:23:36.929328 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.929307 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/premium-w3-da5f0b-kserve-cf7f7f7f7-gm6qd"] Nov 03 22:23:36.994168 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.994140 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv"] Nov 03 22:23:36.996918 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:36.996898 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/distinct-2-w3-da5f0b-kserve-5f45b9c995-wxkzv"] Nov 03 22:23:37.635013 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:37.634982 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:38.504273 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:38.504241 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" path="/var/lib/kubelet/pods/858f0ceb-1bfa-4d79-a108-bff88911fdc2/volumes" Nov 03 22:23:38.504855 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:38.504834 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" path="/var/lib/kubelet/pods/c74eaac2-2350-4ec4-8601-a209a18500ef/volumes" Nov 03 22:23:43.262190 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:43.262156 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:50.390136 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.390107 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4"] Nov 03 22:23:50.390662 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.390280 5931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" podUID="443b17c9-5a0b-496f-a013-a55819000d43" containerName="maas-api" containerID="cri-o://ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52" gracePeriod=30 Nov 03 22:23:50.587147 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.587088 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:50.632520 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.632491 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 22:23:50.749472 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.749414 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443b17c9-5a0b-496f-a013-a55819000d43-kube-api-access-fdlr5\" (UniqueName: \"kubernetes.io/projected/443b17c9-5a0b-496f-a013-a55819000d43-kube-api-access-fdlr5\") pod \"443b17c9-5a0b-496f-a013-a55819000d43\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " Nov 03 22:23:50.749589 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.749491 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls\") pod \"443b17c9-5a0b-496f-a013-a55819000d43\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " Nov 03 22:23:50.749589 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.749527 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443b17c9-5a0b-496f-a013-a55819000d43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443b17c9-5a0b-496f-a013-a55819000d43-openshift-service-ca\") pod \"443b17c9-5a0b-496f-a013-a55819000d43\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " Nov 03 22:23:50.749589 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.749550 5931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-metrics-certs\") pod \"443b17c9-5a0b-496f-a013-a55819000d43\" (UID: \"443b17c9-5a0b-496f-a013-a55819000d43\") " Nov 03 22:23:50.750058 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.750031 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443b17c9-5a0b-496f-a013-a55819000d43-openshift-service-ca" pod "443b17c9-5a0b-496f-a013-a55819000d43" (UID: "443b17c9-5a0b-496f-a013-a55819000d43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:50.751661 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.751605 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls" pod "443b17c9-5a0b-496f-a013-a55819000d43" (UID: "443b17c9-5a0b-496f-a013-a55819000d43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:50.751763 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.751666 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443b17c9-5a0b-496f-a013-a55819000d43-kube-api-access-fdlr5" pod "443b17c9-5a0b-496f-a013-a55819000d43" (UID: "443b17c9-5a0b-496f-a013-a55819000d43"). InnerVolumeSpecName "kube-api-access-fdlr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:50.752099 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.752079 5931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-metrics-certs" pod "443b17c9-5a0b-496f-a013-a55819000d43" (UID: "443b17c9-5a0b-496f-a013-a55819000d43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:50.794749 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.794725 5931 generic.go:356] "Generic (PLEG): container finished" podID="443b17c9-5a0b-496f-a013-a55819000d43" containerID="ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52" exitCode=0 Nov 03 22:23:50.794853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.794758 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" event={"ID":"443b17c9-5a0b-496f-a013-a55819000d43","Type":"ContainerDied","Data":"ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52"} Nov 03 22:23:50.794853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.794775 5931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" Nov 03 22:23:50.794853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.794826 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4" event={"ID":"443b17c9-5a0b-496f-a013-a55819000d43","Type":"ContainerDied","Data":"300b3f79091e382fd647fca130184846692f77c2f6161021a94a2b7da82ad86f"} Nov 03 22:23:50.794853 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.794847 5931 scope.go:122] "RemoveContainer" containerID="ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52" Nov 03 22:23:50.803919 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.803902 5931 scope.go:122] "RemoveContainer" containerID="ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52" Nov 03 22:23:50.804195 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:23:50.804176 5931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52\": container with ID starting with ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52 not found: ID does not exist" containerID="ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52" Nov 03 22:23:50.804255 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.804202 5931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52"} err="failed to get container status \"ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52\": rpc error: code = NotFound desc = could not find container \"ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52\": container with ID starting with ab1cee736dc853d5c39d5088f5f89da95a8408e48366807f33ef8b008fd77a52 not found: ID does not exist" Nov 03 22:23:50.850682 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.850653 5931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdlr5\" (UniqueName: \"kubernetes.io/projected/443b17c9-5a0b-496f-a013-a55819000d43-kube-api-access-fdlr5\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.850682 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.850679 5931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-maas-api-tls\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.850812 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.850695 5931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443b17c9-5a0b-496f-a013-a55819000d43-openshift-service-ca\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.850812 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.850729 5931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443b17c9-5a0b-496f-a013-a55819000d43-metrics-certs\") on node \"ip-10-0-128-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.955318 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.955293 5931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4"] Nov 03 22:23:50.958901 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:50.958880 5931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-da5f0b-5dc9795bdf-5cgd4"] Nov 03 22:23:52.073217 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:52.073189 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:52.351718 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:52.351687 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:52.504262 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:52.504226 5931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443b17c9-5a0b-496f-a013-a55819000d43" path="/var/lib/kubelet/pods/443b17c9-5a0b-496f-a013-a55819000d43/volumes" Nov 03 22:23:53.023860 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:53.023828 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:54.204322 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:54.204291 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:54.433110 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:54.433084 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:58.466891 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:58.466829 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:58.707451 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:58.707413 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:58.941866 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:58.941841 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:59.373055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:59.373027 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:59.703861 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:59.703800 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:59.931744 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:23:59.931718 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:00.158687 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:00.158656 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:00.998588 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:00.998543 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-585df4d986-v4wk5_02d38050-82f2-4ec5-a746-2a442440d913/authorino/0.log" Nov 03 22:24:03.067098 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:24:03.067064 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:24:04.334467 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:04.334411 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:07.632957 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:07.632930 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-585df4d986-v4wk5_02d38050-82f2-4ec5-a746-2a442440d913/authorino/0.log" Nov 03 22:24:25.588233 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:25.588200 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-585df4d986-v4wk5_02d38050-82f2-4ec5-a746-2a442440d913/authorino/0.log" Nov 03 22:24:29.649819 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:29.649791 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545b54f677-jpckr_3b8a368e-ca24-45ba-b760-500cab1484f4/kube-auth-proxy/0.log" Nov 03 22:24:29.889774 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:29.889750 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-b4xql_5a40b3c2-8230-4946-a555-132a0de19452/istio-proxy/0.log" Nov 03 22:24:30.372089 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:30.372062 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-42bpn_8c8c4e07-e240-416d-a7ca-ddf5c91b9d54/payload-pre-processing/0.log" Nov 03 22:24:30.577486 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:30.577464 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-chklw_20bf40f5-8b81-478e-9e5b-c0fa554aad05/payload-processing/0.log" Nov 03 22:24:31.688276 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:31.688246 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-d9hbz_738cd3d2-8890-421b-bb39-38f62d78d853/main/0.log" Nov 03 22:24:32.172232 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:32.172209 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-z7m95_462249ac-4a5a-486f-a417-553aa37c719b/main/0.log" Nov 03 22:24:32.894224 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:32.894198 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hxsvp_fdf84aaa-c653-42c2-9c51-eeca59ecee33/main/0.log" Nov 03 22:24:33.066895 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:24:33.066866 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:24:33.391920 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:33.391892 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-gh74l_d2e0fd94-4550-4065-b573-db223a11f0ac/main/0.log" Nov 03 22:24:43.974390 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974314 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ph99f/must-gather-rdz2b"] Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974693 5931 cpu_manager.go:414] "containerMap: removing container" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974706 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974718 5931 cpu_manager.go:414] "containerMap: removing container" podUID="443b17c9-5a0b-496f-a013-a55819000d43" containerName="maas-api" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974723 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443b17c9-5a0b-496f-a013-a55819000d43" containerName="maas-api" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974734 5931 cpu_manager.go:414] "containerMap: removing container" podUID="15288224-fbef-47db-b0a5-8d4f488a6915" containerName="payload-pre-processing" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974739 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15288224-fbef-47db-b0a5-8d4f488a6915" containerName="payload-pre-processing" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974747 5931 cpu_manager.go:414] "containerMap: removing container" podUID="d690c88c-dbed-4959-9c2c-722a6f41e8b6" containerName="payload-processing" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974752 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d690c88c-dbed-4959-9c2c-722a6f41e8b6" containerName="payload-processing" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974758 5931 cpu_manager.go:414] "containerMap: removing container" podUID="09f06984-5f02-4e02-97fb-6d63c69fca38" containerName="payload-pre-processing" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974763 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f06984-5f02-4e02-97fb-6d63c69fca38" containerName="payload-pre-processing" Nov 03 22:24:43.974769 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974773 5931 cpu_manager.go:414] "containerMap: removing container" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974777 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974784 5931 cpu_manager.go:414] "containerMap: removing container" podUID="59960395-b38c-4814-8fa6-8918fb2193f6" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974788 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59960395-b38c-4814-8fa6-8918fb2193f6" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974795 5931 cpu_manager.go:414] "containerMap: removing container" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974799 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974807 5931 cpu_manager.go:414] "containerMap: removing container" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974811 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974822 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974825 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974832 5931 cpu_manager.go:414] "containerMap: removing container" podUID="549c57f9-0b18-48da-9712-73248d9d5cb8" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974836 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549c57f9-0b18-48da-9712-73248d9d5cb8" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974842 5931 cpu_manager.go:414] "containerMap: removing container" podUID="ecc6892b-cda8-479d-9c04-b5d16efd8b4a" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974847 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc6892b-cda8-479d-9c04-b5d16efd8b4a" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974853 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6ae84c73-222d-40e4-ba77-e046ef07b02f" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974857 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae84c73-222d-40e4-ba77-e046ef07b02f" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974861 5931 cpu_manager.go:414] "containerMap: removing container" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974866 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974872 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974877 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974882 5931 cpu_manager.go:414] "containerMap: removing container" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974886 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974892 5931 cpu_manager.go:414] "containerMap: removing container" podUID="bd483f46-2923-4f02-bfe1-6fe9ad32d374" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974896 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd483f46-2923-4f02-bfe1-6fe9ad32d374" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974902 5931 cpu_manager.go:414] "containerMap: removing container" podUID="85227cfc-1db8-46e0-964a-c24442389ad2" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974906 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85227cfc-1db8-46e0-964a-c24442389ad2" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974911 5931 cpu_manager.go:414] "containerMap: removing container" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974915 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974921 5931 cpu_manager.go:414] "containerMap: removing container" podUID="d2ca03b3-6b7c-4160-a9a1-9ebf98009784" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974926 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ca03b3-6b7c-4160-a9a1-9ebf98009784" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974935 5931 cpu_manager.go:414] "containerMap: removing container" podUID="6f745fb4-1903-4e95-b3c9-92429f5400d5" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974939 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f745fb4-1903-4e95-b3c9-92429f5400d5" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974944 5931 cpu_manager.go:414] "containerMap: removing container" podUID="18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974948 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974956 5931 cpu_manager.go:414] "containerMap: removing container" podUID="c30da697-3933-4894-8f70-da325802d8b6" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974960 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30da697-3933-4894-8f70-da325802d8b6" containerName="main" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974965 5931 cpu_manager.go:414] "containerMap: removing container" podUID="0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974970 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974976 5931 cpu_manager.go:414] "containerMap: removing container" podUID="cc07e711-7215-44f0-86b5-69a682869a4d" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974980 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc07e711-7215-44f0-86b5-69a682869a4d" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974988 5931 cpu_manager.go:414] "containerMap: removing container" podUID="f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974992 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.974998 5931 cpu_manager.go:414] "containerMap: removing container" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975001 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975009 5931 cpu_manager.go:414] "containerMap: removing container" podUID="2a656523-f835-437b-80d2-fff51c87f061" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975013 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a656523-f835-437b-80d2-fff51c87f061" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975019 5931 cpu_manager.go:414] "containerMap: removing container" podUID="fc52c97f-0de4-4af8-acb3-7da8534e527c" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975023 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc52c97f-0de4-4af8-acb3-7da8534e527c" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975030 5931 cpu_manager.go:414] "containerMap: removing container" podUID="3addd93b-3e66-469f-bd37-267517de0347" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975034 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3addd93b-3e66-469f-bd37-267517de0347" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975041 5931 cpu_manager.go:414] "containerMap: removing container" podUID="479afc9c-4f61-40bb-8b23-5de937c635c3" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975044 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479afc9c-4f61-40bb-8b23-5de937c635c3" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975052 5931 cpu_manager.go:414] "containerMap: removing container" podUID="b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975056 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975062 5931 cpu_manager.go:414] "containerMap: removing container" podUID="13d47dae-3cc7-4a35-87cb-fde3815aa4f0" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975066 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d47dae-3cc7-4a35-87cb-fde3815aa4f0" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975072 5931 cpu_manager.go:414] "containerMap: removing container" podUID="72f4508e-ba50-469e-b9c6-60c528f613bc" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975076 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f4508e-ba50-469e-b9c6-60c528f613bc" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975082 5931 cpu_manager.go:414] "containerMap: removing container" podUID="8a6b97c4-9953-4b74-acbe-01ecf1043be4" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975088 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6b97c4-9953-4b74-acbe-01ecf1043be4" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975139 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95cb44f-4d6a-48d6-a0a2-6e912d5ff997" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975146 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975156 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a656523-f835-437b-80d2-fff51c87f061" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975163 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc52c97f-0de4-4af8-acb3-7da8534e527c" containerName="payload-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975170 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3addd93b-3e66-469f-bd37-267517de0347" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975177 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="479afc9c-4f61-40bb-8b23-5de937c635c3" containerName="maas-api" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975183 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74de4ef-bbf0-4916-bdb3-f2761fdf4abb" containerName="payload-pre-processing" Nov 03 22:24:43.975159 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975188 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d47dae-3cc7-4a35-87cb-fde3815aa4f0" containerName="maas-api" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975193 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f4508e-ba50-469e-b9c6-60c528f613bc" containerName="payload-pre-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975200 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975210 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6b97c4-9953-4b74-acbe-01ecf1043be4" containerName="maas-api" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975218 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="858f0ceb-1bfa-4d79-a108-bff88911fdc2" containerName="main" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975226 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="443b17c9-5a0b-496f-a013-a55819000d43" containerName="maas-api" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975231 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="15288224-fbef-47db-b0a5-8d4f488a6915" containerName="payload-pre-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975237 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d690c88c-dbed-4959-9c2c-722a6f41e8b6" containerName="payload-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975242 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f06984-5f02-4e02-97fb-6d63c69fca38" containerName="payload-pre-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975247 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="005066fa-ff22-419b-af08-6f93eb95ce5c" containerName="main" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975252 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="59960395-b38c-4814-8fa6-8918fb2193f6" containerName="payload-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975258 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975263 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975268 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="549c57f9-0b18-48da-9712-73248d9d5cb8" containerName="payload-pre-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975274 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc6892b-cda8-479d-9c04-b5d16efd8b4a" containerName="payload-pre-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975278 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae84c73-222d-40e4-ba77-e046ef07b02f" containerName="maas-api" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975282 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975287 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74eaac2-2350-4ec4-8601-a209a18500ef" containerName="main" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975291 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="206bb57e-8add-4c82-be28-d47efff96d09" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975295 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd483f46-2923-4f02-bfe1-6fe9ad32d374" containerName="payload-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975300 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="85227cfc-1db8-46e0-964a-c24442389ad2" containerName="payload-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975304 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975309 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="81838a1c-e57a-49e0-9286-5340e392a979" containerName="main" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975316 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ca03b3-6b7c-4160-a9a1-9ebf98009784" containerName="payload-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975320 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f745fb4-1903-4e95-b3c9-92429f5400d5" containerName="payload-pre-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975325 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="18dbaa83-a7a0-4b87-8e91-cb0b02c5e67a" containerName="payload-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975330 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30da697-3933-4894-8f70-da325802d8b6" containerName="main" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975334 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0118c7bd-d838-4fb3-bb2a-f6f6aad0be9f" containerName="payload-pre-processing" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975339 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc07e711-7215-44f0-86b5-69a682869a4d" containerName="maas-api" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975419 5931 cpu_manager.go:414] "containerMap: removing container" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975425 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c1ac5d-4215-4817-af62-4931f675c3db" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975433 5931 cpu_manager.go:414] "containerMap: removing container" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975437 5931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06145d74-f6d7-4895-8e8e-ad0582b1f784" containerName="istio-proxy" Nov 03 22:24:43.977311 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.975502 5931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c27741f-5733-4eae-92a1-40eb8fc93976" containerName="istio-proxy" Nov 03 22:24:43.978487 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.978267 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:43.996689 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:43.996645 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:44.034811 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.034782 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/must-gather-rdz2b"] Nov 03 22:24:44.067404 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.067384 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjj6\" (UniqueName: \"kubernetes.io/projected/9f237350-fcda-451d-a13f-4cec3083f9d7-kube-api-access-cqjj6\") pod \"must-gather-rdz2b\" (UID: \"9f237350-fcda-451d-a13f-4cec3083f9d7\") " pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:44.067506 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.067437 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f237350-fcda-451d-a13f-4cec3083f9d7-must-gather-output\") pod \"must-gather-rdz2b\" (UID: \"9f237350-fcda-451d-a13f-4cec3083f9d7\") " pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:44.082143 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.082117 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ph99f\"/\"openshift-service-ca.crt\"" Nov 03 22:24:44.082260 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.082204 5931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ph99f\"/\"default-dockercfg-vjzx4\"" Nov 03 22:24:44.082260 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.082211 5931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ph99f\"/\"kube-root-ca.crt\"" Nov 03 22:24:44.167847 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.167823 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f237350-fcda-451d-a13f-4cec3083f9d7-must-gather-output\") pod \"must-gather-rdz2b\" (UID: \"9f237350-fcda-451d-a13f-4cec3083f9d7\") " pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:44.167951 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.167873 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjj6\" (UniqueName: \"kubernetes.io/projected/9f237350-fcda-451d-a13f-4cec3083f9d7-kube-api-access-cqjj6\") pod \"must-gather-rdz2b\" (UID: \"9f237350-fcda-451d-a13f-4cec3083f9d7\") " pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:44.168123 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.168107 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f237350-fcda-451d-a13f-4cec3083f9d7-must-gather-output\") pod \"must-gather-rdz2b\" (UID: \"9f237350-fcda-451d-a13f-4cec3083f9d7\") " pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:44.224024 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.224006 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjj6\" (UniqueName: \"kubernetes.io/projected/9f237350-fcda-451d-a13f-4cec3083f9d7-kube-api-access-cqjj6\") pod \"must-gather-rdz2b\" (UID: \"9f237350-fcda-451d-a13f-4cec3083f9d7\") " pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:44.283864 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.283765 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:44.291746 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.291729 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/must-gather-rdz2b" Nov 03 22:24:44.661545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.661519 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/must-gather-rdz2b"] Nov 03 22:24:44.664436 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:24:44.664409 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f237350_fcda_451d_a13f_4cec3083f9d7.slice/crio-6631ecd68776d848214a47699dcce150fb189272a5a799a78409b400553db902 WatchSource:0}: Error finding container 6631ecd68776d848214a47699dcce150fb189272a5a799a78409b400553db902: Status 404 returned error can't find the container with id 6631ecd68776d848214a47699dcce150fb189272a5a799a78409b400553db902 Nov 03 22:24:44.666055 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:44.666036 5931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:24:45.062408 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:45.062334 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/must-gather-rdz2b" event={"ID":"9f237350-fcda-451d-a13f-4cec3083f9d7","Type":"ContainerStarted","Data":"6631ecd68776d848214a47699dcce150fb189272a5a799a78409b400553db902"} Nov 03 22:24:46.070674 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:46.070604 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/must-gather-rdz2b" event={"ID":"9f237350-fcda-451d-a13f-4cec3083f9d7","Type":"ContainerStarted","Data":"486f424e6a2aa912669b73419767090d71a9e36542c88512298f56cadd0adb62"} Nov 03 22:24:47.079945 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:47.079910 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/must-gather-rdz2b" event={"ID":"9f237350-fcda-451d-a13f-4cec3083f9d7","Type":"ContainerStarted","Data":"826da5643514aa5ec4703d964bdb237d1deb07dce6d0f3878e4625ef0d36d9b7"} Nov 03 22:24:47.193789 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:47.193733 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ph99f/must-gather-rdz2b" podStartSLOduration=3.012017149 podStartE2EDuration="4.193716636s" totalImagesPullingTime="1.181699487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:47.19273319 +0000 UTC m=+4154.803581015" watchObservedRunningTime="2026-11-03 22:24:47.193716636 +0000 UTC m=+4154.804564451" Nov 03 22:24:52.729031 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:52.729002 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9vfbk_d8acde6b-bb34-4c16-9afe-aa4c21f59dd6/global-pull-secret-syncer/0.log" Nov 03 22:24:55.793174 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:55.793142 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" Nov 03 22:24:57.413435 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:57.413405 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-179.us-west-1.compute.internal_1745eaf53680bc2ae777ad5a0ee0f5c6/haproxy/0.log" Nov 03 22:24:59.722048 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:24:59.722017 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:03.067525 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:25:03.067485 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:25:19.050538 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:19.050501 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:19.294589 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:19.294559 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:19.622118 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:19.622086 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545b54f677-jpckr_3b8a368e-ca24-45ba-b760-500cab1484f4/kube-auth-proxy/0.log" Nov 03 22:25:19.839098 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:19.839052 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-b4xql_5a40b3c2-8230-4946-a555-132a0de19452/istio-proxy/0.log" Nov 03 22:25:20.260865 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:20.260802 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-42bpn_8c8c4e07-e240-416d-a7ca-ddf5c91b9d54/payload-pre-processing/0.log" Nov 03 22:25:20.473373 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:20.473342 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-chklw_20bf40f5-8b81-478e-9e5b-c0fa554aad05/payload-processing/0.log" Nov 03 22:25:21.973932 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:21.973899 5931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm"] Nov 03 22:25:21.977253 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:21.977226 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.007206 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.007179 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-sys\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.007350 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.007229 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-podres\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.007350 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.007330 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-lib-modules\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.007471 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.007440 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgggl\" (UniqueName: \"kubernetes.io/projected/943f6902-a4f2-4bc8-8280-515be421a3e4-kube-api-access-vgggl\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.007534 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.007482 5931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-proc\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.035014 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.034976 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm"] Nov 03 22:25:22.107994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.107959 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgggl\" (UniqueName: \"kubernetes.io/projected/943f6902-a4f2-4bc8-8280-515be421a3e4-kube-api-access-vgggl\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.107994 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.107999 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-proc\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.108267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.108041 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-sys\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.108267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.108067 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-podres\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.108267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.108098 5931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-lib-modules\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.108267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.108172 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-sys\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.108267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.108204 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-podres\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.108267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.108210 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-lib-modules\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.108267 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.108255 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/943f6902-a4f2-4bc8-8280-515be421a3e4-proc\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.164588 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.164556 5931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgggl\" (UniqueName: \"kubernetes.io/projected/943f6902-a4f2-4bc8-8280-515be421a3e4-kube-api-access-vgggl\") pod \"perf-node-gather-daemonset-kgsvm\" (UID: \"943f6902-a4f2-4bc8-8280-515be421a3e4\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.291570 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.291477 5931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:22.665230 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:22.665191 5931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm"] Nov 03 22:25:22.670082 ip-10-0-128-179 kubenswrapper[5931]: W1103 22:25:22.670050 5931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod943f6902_a4f2_4bc8_8280_515be421a3e4.slice/crio-7fc8f12fa3ef6098852cb990696deb12d18b9d57b874e45c1922840fa012096e WatchSource:0}: Error finding container 7fc8f12fa3ef6098852cb990696deb12d18b9d57b874e45c1922840fa012096e: Status 404 returned error can't find the container with id 7fc8f12fa3ef6098852cb990696deb12d18b9d57b874e45c1922840fa012096e Nov 03 22:25:23.301316 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:23.301279 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" event={"ID":"943f6902-a4f2-4bc8-8280-515be421a3e4","Type":"ContainerStarted","Data":"8e3eb2211aa0c7e11ef606fa8e6825cd648a34a71d22c7bfc6d216ced0234a14"} Nov 03 22:25:23.302284 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:23.301342 5931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" event={"ID":"943f6902-a4f2-4bc8-8280-515be421a3e4","Type":"ContainerStarted","Data":"7fc8f12fa3ef6098852cb990696deb12d18b9d57b874e45c1922840fa012096e"} Nov 03 22:25:23.302284 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:23.301459 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:23.415440 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:23.415379 5931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" podStartSLOduration=2.415362759 podStartE2EDuration="2.415362759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:23.414483486 +0000 UTC m=+4191.025331300" watchObservedRunningTime="2026-11-03 22:25:23.415362759 +0000 UTC m=+4191.026210573" Nov 03 22:25:26.839545 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:26.839517 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-585df4d986-v4wk5_02d38050-82f2-4ec5-a746-2a442440d913/authorino/0.log" Nov 03 22:25:28.954250 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:28.954216 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:29.231032 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:29.230957 5931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ljk7_02c1435f-0c51-4647-b1c2-0334a39a1af2/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:29.316101 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:29.316073 5931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-kgsvm" Nov 03 22:25:33.074673 ip-10-0-128-179 kubenswrapper[5931]: E1103 22:25:33.074635 5931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcf7l" containerName="extractor" Nov 03 22:25:35.921750 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:35.921722 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:25:35.927862 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:35.927836 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8bjk_4f4cad3e-0d13-4459-b970-79675d07443d/ovn-acl-logging/0.log" Nov 03 22:25:43.156496 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:43.156459 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-dnp2h_37f406f3-d2fd-44b6-8cb7-b66aa291772b/cert-manager-controller/0.log" Nov 03 22:25:43.368730 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:43.368696 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-wv8jv_2325c2a0-e613-49b7-8093-0b7813636a63/cert-manager-cainjector/0.log" Nov 03 22:25:43.580507 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:43.580434 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-qcm8d_bc35cc1e-3a8b-423a-ada5-34d99cc05973/cert-manager-webhook/0.log" Nov 03 22:25:46.143730 ip-10-0-128-179 kubenswrapper[5931]: I1103 22:25:46.143693 5931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545b54f677-jpckr_3b8a368e-ca24-45ba-b760-500cab1484f4/kube-auth-proxy/0.log"